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ABSTRACT 


This  thesis  presents  and  analyses  the  results  of  a  survey  designed  to  determine  how 
Department  of  Defense  financial  management  education  and  training  programs  assess  the  quality  of 
their  programs.  Quality  in  the  context  of  this  thesis  means  providing  accurate,  valid,  comprehensive 
and  up-to-date  information  to  meet  the  needs  of  clients  and  customers.  The  thesis  explains  the  need 
within  the  Department  of  Defense  for  financial  management  education.  It  documents  the  methodology 
employed  in  developing  the  survey.  The  thesis  discusses  the  general  characteristics  of  financial 
management  education  and  training  programs  and  summarizes  the  methods  employed  to  ensure  the 
quality  of  these  programs. 
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I.  INTRODUCTION 

A.  BACKGROUND 

Financial  management  is  one  of  the  most  Important 
functional  tasks  performed  in  the  Department  of  Defense  (DoD)  . 
With  an  annual  budget  in  the  range  of  $  290  billion,  the 
development  of  detailed  budgets  and  plans,  day  to  day 
execution,  and  accounting  for  the  proper  expenditure  of  these 
is  a  major  enterprise. 

A  study  conducted  by  the  Defense  Manpower  Data  Center 
(DMDC)  in  December,  1990  indicated  that  there  were 
approximately  119,000  personnel  working  in  or  qualified  to 
perform  financial  management  related  tasks  within  DoD.  This 
is  another  indication  of  the  magnitude  of  the  effort  dedicated 
to  financial  management. 

The  interest  of  DoD  leadership  in  the  financial  management 
field  is  not  a  recent  development  and  can  be  traced  back  prior 
to  the  mid  -  1800' s.  However,  several  recent  developments 
have  increased  the  awareness  within  DoD  of  the  in^ortance  of 
the  financial  manager.  One  of  these  developments  was  the 
legislation  calling  for  Chief  Financial  Officers  (CFO's)  to  be 
appointed  throughout  the  Federal  Government.  Another  factor 
has  been  the  relative  decline  in  federal  budgetary  support  for 
the  military,  thus  emphasizing  the  need  to  carefully  budget 
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and  spend  scarce  resources.  Yet  another  factor  were  the 
Defense  Management  Report  Decisions  (DMRD)  985  for  fiscal 
years  1991  and  1992.  These  DMRD's  focused  renewed  attention  on 
the  financial  management  community  throughout  DoD.  To  briefly 
quote  from  the  first  DMRD: 

The  financial  management  community  is  entering  an  era  of 
new  technology,  major  and  rapid  change  in  the  domestic  and 
international  environments,  and  decreased  resources  for 
defense.  In  the  future  we  will  have  to  work  smarter,  more 
cost  effectively,  and  to  respond  more  quickly  than  ever 
before.  More  effective  education  and  training  Is  pivotal 
If  the  department  Is  to  answer  this  challenge.  (Eit^hasis 
mine) 

In  light  of  the  concern  over  financial  management  within 
DoD,  one  of  the  questions  that  arose  was  -  how  do  those 
agencies  and  departments  providing  financial  management 
education  and  training  courses  and  programs  ensure  the  quality 
of  their  offering?  This  research  is  a  direct  result  of  the 
concern  for  quality  assessment  of  financial  managt-Tient . 

B .  PURPOSE 

The  purpose  of  this  research  is  to  identify  the  quality 
assessment  methods  used  by  providers  of  financial  management 
education  and  training  courses  and  programs  offered  throughout 
the  DoD.  The  providers  of  financial  management  education  and 
training  courses  and  programs  were  asked  to  complete  a 
detailed  questionnaire  describing  their  programmatic 
offerings.  The  survey  was  used  as  a  vehicle  to  collect 
detailed  information  on  methods  employed  by  the  various 
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agencies  and  departments  to  ensure  the  effectiveness  of  their 
courses  and  programs.  Finally,  the  study  compares  the 
various  assessment  technicjues  to  determine  what  commonalities 
exist  with  respect  to  quality  assessment.  Similarly,  the 
research  will  examine  unique  characteristics,  if  any,  of 
quality  assessment  methods  employed  in  these  agencies  and 
departments . 

An  important  by-product  of  the  research  is  the  development 
of  a  comprehensive  data  base  of  information  on  quality 
assessment  used  by  all  providers  of  financial  management 
education  and  training  in  DoD. 

C.  RESEARCH  QUESTIONS 

The  research  questions  examined  in  this  study  are: 

•  What  financial  management  courses  and  programs  are  offered 
within  DoD  and  what  are  the  general  characteristics  of 
these  courses  and  programs? 

•  What  methods  are  employed  to  ensure  the  quality  of 
financial  management  courses  and  programs  within  DoD? 


D.  SCOPE 

The  major  emphasis  of  this  research  is  to  collect  current 
information  on  quality  assessment  measures  employed  by 
providers  of  financial  management  and  education  programs  and 
courses  within  DoD.  To  accomplish  this  task,  all  providers  of 
financial  management  education  and  training  are  identified. 
Once  this  comprehensive  list  of  providers  was  obtained,  these 
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agencies  and  departments  were  queried  as  to  the  methods 
employed  to  ensure  the  quality  of  their  programs. 

All  military  departments  and  major  consonants  of  DoD  such 
as  the  Defense  Logistics  Agency  and  the  Defense  Finance  and 
Accounting  Service  are  included  in  the  study.  In-house  as 
well  as  outside  contracted  providers  also  are  included  in  the 
study.  Civilian  as  well  as  military  financial  management 
education  and  training  programs  are  included.  Specifically 
excluded  from  this  research  are  financial  management  education 
and  training  programs  based  on  the  "correspondence"  method. 

B.  METHODOLOGY 

The  methodology  for  conducting  this  research  involved  four 
distinct  steps.  Initially,  the  research  concentrated  on  a 
through  review  of  current  literature,  instructions, 
directives,  reference  materials  and  guidance  dealing  with 
financial  management  education  and  training. 

Secondly,  the  development  of  a  comprehensive  survey 
instrument  was  undertaken.  Survey  methodology  in  the 
literature  was  reviewed.  Prior  surveys  used  by  graduate 
students  at  the  Naval  Postgraduate  School  (NPS)  also  were 
reviewed  for  format  and  answer  criteria.  Professors  and  other 
professionals  in  the  field  of  education  assessment  were 
queried  on  appropriate  survey  questions.  A  telephone  poll  was 
conducted  with  providers  of  financial  management  education  and 
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training  to  elicit  further  areas  of  concern  and  to  develop 
further  questions  for  inclusion  in  the  survey. 

To  verify  the  contents  and  organization  of  the 
questionnaire,  a  pilot  test  was  conducted  with  two  providers 
of  financial  management  education  and  training. 

Thirdly,  the  finished  survey  questionnaire  was  mailed  to 
all  identified  providers  of  financial  management  education  and 
training  within  DoD. 

Finally,  the  survey  responses  were  analyzed. 

Further  discussion  of  key  areas  of  the  research 
methodology  employed  in  this  report  are  presented  below. 

1.  Literature  Review 

The  initial  sources  of  data  for  this  thesis  were 
developed  through  a  comprehensive  literature  search.  The 
search  focused  on  three  distinct  areas.  First,  the  field  of 
survey  methodology  was  reviewed  to  determine  the  format  and 
style  of  the  questionnaire  that  would  be  used  to  gather  the 
primary  information  upon  which  this  thesis  is  based.  This 
portion  of  the  review  included  careful  attention  to  the  design 
of  survey  questions,  formats  of  surveys  and  the  method  by 
which  the  data  was  to  be  collected,  e.g.,  telephone  interviews 
or  mailed  surveys. 

A  detailed  examination  was  also  conducted  of  actual 
surveys  used  by  graduate  students  at  the  Naval  Postgraduate 

I 

School,  surveys  employed  by  educational  professionals  and 
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general  purpose  surveys  which  were  received  by  various 
professors  at  NPS. 

The  second  area  of  literature  research  focused  on 
financial  management  within  DoD.  Information  specifically 
concerned  with  financial  management  training  course 
availability,  course  content,  and  targeted  populations  was 
reviewed.  Applicable  instructions  and  guidance  for  financial 
management  personnel  were  also  reviewed.  Finally,  research 
results  and  recommendations  developed  in  prior  studies  of 
financial  management  were  reviewed. 

Thirdly,  relevant  literature  on  quality  assessment  and 
assessment  techniques  was  reviewed. 

2 .  Telephone  Interviews 

To  help  ensure  a  relevant  and  comprehensive  survey 
questionnaire  was  prepared,  a  telephone  interview  was 
conducted  with  administrators  and  professionals  employed  by 
providers  of  financial  management  education  and  training 
within  DoD.  Appendix  A  list  those  individuals  and  providers 
contacted  in  this  process. 

3 .  Survey  Questionnaire 

A  questionnaire  (Appendix  B)  was  distributed  to  all 
agencies  and  departments  within  DoD  providing  financial 
management  education  and  training.  These  agencies  and 
departments  were  identified  by  using  the  DoD  Training  and 
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Performance  Data  Center's  listing  of  financial  management 
education  and  training  providers. 

The  questionnaire  was  designed  by  a  team  of  Naval 
Postgraduate  School  faculty  and  the  author  to  assess  how  these 
providers  ensure  the  quality  of  their  courses  and  programs. 
Appendix  C  lists  those  agencies  that  were  contacted  and 
responded  to  the  survey.  Appendix  D  lists  those  agencies  that 
failed  to  respond  to  the  survey. 

F.  ORGANIZATION  OF  THE  STUDY 

This  thesis  is  organized  into  five  chapters.  Chapter  I 
provides  a  general  introduction  to  the  area  of  study.  It 
includes  a  brief  background  statement,  purpose  of  the 
research,  detailed  research  questions  to  be  examined,  scope  of 
the  study,  and  methodology  utilized  in  the  research  effort. 
Chapter  II  provides  background  information  concerning  the 
financial  management  community,  DoD  concerns  with  the 
"effectiveness"  or  quality  of  financial  management  education 
and  training,  and  a  brief  review  of  quality  assessment 
efforts.  Chapter  III  details  the  methodology  employed  in 
developing  the  survey  instrument  and  collecting  the  completed 
surveys.  Chapter  IV  analyses  the  results  of  the  surveys  in 
terms  of  the  methods  employed  to  ensure  quality.  Chapter  V 
presents  the  conclusions  reached  as  a  result  of  the  research. 
The  chapter  also  provides  answers  to  the  research  questions 
examined  in  the  thesis. 
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II.  FINANCIAL  MANAGEMENT 


A.  INTRODUCTION 

Management  of  the  federal  government's  financial  resources 
has  become  one  of  the  main  issues  in  the  debate  over  the 
growing  size  of  the  federal  budget  and  the  budget  deficit. 
Concern  over  these  issues  reaches  from  the  President, 
congressional  leadership,  and  the  general  public  to  the 
leadership  of  our  military  forces.  Efficient  and  effective 
accounting,  budgeting  and  expenditure  of  public  funds  have 
become  the  watchwords  of  both  the  political  and  military 
leadership  within  DoD. 

[Ref.  1;  pp.  1-8] 

DoD  can  be  compared  to  the  largest  corporations  in  the 
United  States.  Total  employment  exceeds  2.6  million 
personnel  of  which  approximately  1.6  million  are  uniformed 
service  personnel  with  the  balance  being  civilian  personnel. 
Budget  authority  for  the  1992  fiscal  year  budget  for  DoD  is  $ 
290.9  billion,  larger  by  far  than  any  corporate  budget. 

During  the  Presidency  of  Ronald  Reagan,  the  defense  budget 
increased  considerably.  And  despite  the  recent  budgetary 
decline  (in  constant  dollars) ,  the  need  for  effective 
financial  management  has  not  diminished.  In  the  current  era 
of  military  force  structure  reductions  and  concomitant 
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budgetary  reductions,  DoD  must  ensure  that  adequate  resources 
are  available  and  managed  wisely  and  efficiently  to  meet  the 
defense  requirements  of  the  United  States.  The  relative 
decline  in  federal  support  for  the  Defense  Department  is 
detailed  in  Table  I.  Whether  in  constant  dollars  or 
discounted  for  inflation,  the  decline  in  funding  available  to 
the  military  is  clear.  [Ref.  2:  p.  A- 6] 

TABLE  I.  PROJECTED  DoD  BUDGET  OUTLAYS  FOR  PY  92-97 


PY 

1992 

1993 

1994 

1995 

1996 

1997 

Cum  % 
Change 

Constant  $ 

294.6 

277.9 

269.4 

268.7 

271.0 

273.6 

(7.1) 

Adjusted  $ 

Note* 

294.6 

266.8 

247.8 

237.9 

228.4 

219.3 

(27.6) 

Source:  The  President's  Budget  for  Fiscal  Year  1993 


To  meet  the  ever-growing  challenge  of  decreasing  financial 
resources  ,  the  financial  management  community  must  operate  at 
peak  efficiency.  Education  and  training  programs  must  be 
designed  to  develop  financial  managers  capable  of  dealing  with 
the  challenges  brought  about  by  reduced  budgets. 
Additionally,  those  education  and  training  resources  must  be 
organized  to  ensure  that  the  most  efficient  and  effective 
financial  management  "corp"  is  available  to  DoD.  The  recent 
legislation  mandating  Chief  Financial  Officers  (CFO's)  for  all 

'  The  adjusted  budgetary  outlays  were  deflated  by  four 
percent  per  year. 
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federal  agencies  and  departments  is  but  one  of  many  example  of 
the  continuing  drive  to  enhance  financial  management  within 
the  Federal  government. 

To  provide  a  brief  guide  to  the  size  of  personnel  devoted 
to  financial  management  within  DoD,  a  study  conducted  in  1990 
by  the  Defense  Manpower  Data  Center  (DMDC)  details  the  number 
of  civilian  and  service  personnel  involved  in 
financial/resource  management.  While  the  aggregate  numbers 
presented  in  Table  II  below  have  been  questioned  by  the 
military  departments  [Ref.  3:  p.l89]  within  DoD  (e.g.,  the 
total  acknowledged  by  these  components  is  approximately  20,000 
less  than  estimated  by  DMDC)  ,  the  n\ambers  generally  represent 
the  total  number  of  personnel  engaged  DoD -wide  in  the 
financial  or  resource  management  field.  Table  II  details  the 
findings  of  the  DMDC  study. 

Whether  the  number  of  personnel  employed  in  financial 
management  is  the  119,000  cited  by  DMDC  or  the  approximate 
100,000  acknowledged  by  individual  DoD  components,  the  total 
manpower  resources  devoted  to  financial/resource  management  is 
impressive.  [Ref.  3:  pp.  187-188] 
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TABLE  II. 


FINANCIAL /RESOURCE  MANAGEMENT  EMPLOYEES  OP  DoD 


CIVILIAN  I  MILITARY 


AGENCY 

PROP /ADMIN 

TECHNICAL 

OFFICER 

ENLISTED 

TOTAL 

ARMY 

27,105 

13,014 

2,746 

3,942 

46,807 

AF 

13,945 

5,647 

1,551 

5,376 

26,519 

NAVY 

15,513 

9,051 

688 

2,872 

28,124 

MC 

921 

1,194 

304 

1,314 

3,733 

DLA 

2,902 

2,671 

5,573 

OTHER 

8,124 

292 

8,416 

TOTAL 

68,510 

31,869 

5,289 

13,504 

119,172 

B.  THE  INCREASED  FOCUS  ON  FINANCIAL  MANAGEMENT 

Financial  management  or  resource  management  has  been 
defined  as  follows: 

Financial/Resource  Management  (F/RM)  is  the 
art  and  science  of  acquiring,  allocating,  and 
controlling  the  use  of  resources  as  expressed 
primarily  in  monetary  terms,  but  also  in  terms 
of  the  physical  resources  themselves  (e.g., 
manpower  and  material) .  F/RM  includes  the 
functions  of  Budget  Formulation,  Development, 
Administration,  Review,  and  Execution; 
Military  and  Civilian  Payroll;  Accounting; 
Auditing;  Payments;  Cost  and  Economic 
Analysis;  Contract  Management  and  Oversight; 
Investment  Management;  Actuarial  Analysis;  and 
Management /Program  Analysis  and  Evaluation. 

This  definition  includes  the 
financial/resource  management  operations 
supporting  deployed  and  battlefield  elements 
with  the  military  units  established  to  provide 
such  support,  [Ref,  3:  p.  7] 
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As  the  above  definition  of  financial  management 
demonstrates,  financial  management  personnel  within  the 
military  perform  a  wide  diversity  of  functions.  A  minimiam  of 
100,000  personnel  within  DoD  are  assigned  to  financial 
management  activities. 

The  need  for  proper  training  and  education  of  financial 
management  personnel  has  been  recognized  by  officials  within 
DoD  but  and  by  others  in  the  federal  government. 

The  Comptroller  of  DoD  noted  the  following: 

The  DoD  Comptroller  is  aware  of  the  increased  importance 
of  education  and  training  for  the  professional  development 
of  the  members  of  the  financial/resource  management  (F/RM) 
community.  Improvements  in  the  effectiveness  and 
efficiency  of  education  and  training  have  become  a  major 
DoD  need.  [Ref.  3:  p.  i] 

The  Federal  Budget  for  Fiscal  Year  1993  also  clearly 
addresses  the  issue  of  financial  management.  As  noted  in  the 
FY  1993  budget,  the  President  requested  $  2.2  billion  for 
financial  management  improvements  throughout  the  Federal 
government,  an  increase  of  $  83  million  over  funds  enacted  for 
FY  1992.  Both  the  Fy  1992  and  Fy  1993  budgets  include 
detailed  proposals  to  "improve  financial  management".  Of 
significance  is  the  following  quote  from  the  FY  1992  budget. 
"Good  management  begins  with  people.  Recruiting  and  retaining 
a  quality  workforce  is  essential  to  ensuring  responsive  and 
effective  services."  [Ref.  4:  p.  303] 

DoD  financial  management  personnel  must  be  proficient  in 
the  basic  fundamental  financial  management  concepts,  policies 
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and  procedures.  They  must  be  able  to  perform  their  jobs 
competently,  effectively  and  efficiently.  Given  the 
heightened  concern  over  financial  management,  DoD  must  provide 
training  and  education  to  financial  management  personnel  that 
ensures  the  achievement  of  these  goals. 

C.  FINANCIAL  MANAGEMENT  EDUCATION  AND  TRAINING 

The  task  of  training  and  educating  the  100,000  or  more 
personnel  involved  in  financial  management  is  a  major 
enterprize.  Appendices  B  and  C  provide  a  complete  listing  of 
the  providers  of  financial  management  education  and  training 
within  DoD.  The  training  and  education  provided  ranges  from 
basic  introductory  courses  in  payroll,  accounting  and 
disbursing  for  junior  enlisted  personnel  to  graduate  level 
Master's  programs  in  financial  management  for  military 
officers  and  mid-grade  civilian  employees. 

A  total  of  29  agencies  and  departments  provide  financial 
management  and  training.  This  number  does  not  include  the 
correspondence  courses  offered  by  the  various  military 
departments  nor  does  it  account  for  education  obtained  by 
military  and  civilian  personnel  outside  of  the  structured 
programs  recognized  in  this  research.  The  diversity  and 
number  of  courses  offered  are  documented  in  Table  III.  [Ref. 
3:  p.  127J 
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TABLE  III.  DoD  FINANCIAL  MANAGEMENT  COURSES  BY  PROVIDER 


Functional  area: 

Army 

Navy 

Air  Force 

DLA 

Total 

Budaet 

9 

2 

13 

Finance 

37 

23 

19 

12 

91 

27 

1 

19 

47 

27 

1 

28 

Totals 

100 

27 

40 

12 

179 

As  Table  III  demonstrates,  the  31  institutions  offer  179 
different  courses  of  instruction.  While  firm  data  are  not 
availcdDle,  it  is  estimated  that  at  least  10  percent  of  the 
financial  management  workforce,  approximately,  10,000 
personnel,  attend  financial  management  education  and  training 
programs  and  courses  each  year.  [Ref.  3] 

D.  RECENT  INITIATIVES  IN  FINANCIAL  MANAGEMENT  EDUCATION  AND 

TRAINING 

The  importance  of  financial  management  and  concomitantly 
the  importance  of  the  education  and  training  of  financial 
managers  has  been  explicitly  recognized  in  the  past  several 
years . 
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1.  Chief  Financial  Officer  Act  of  1990 


One  of  the  more  significant  events  in  the  area  of 
financial  management  education  and  training  was  the  passage  of 
the  Chief  Financial  Officer  Act  of  1990.  That  legislation 
called  for  the  establishment  of  Chief  Financial  Officers 
(CFO's)  for  the  23  major  federal  departments. 

In  addition  to  .establishing  CFO's  for  major 
departments  within  the  Federal  government,  the  Act  created  a 
financial  management  structure  centered  in  the  Office  of 
Management  and  Budget  (0MB) .  Within  0MB,  a  Deputy  Director 
for  Management,  a  Con^troller  and  the  Office  of  Federal 
Financial  Management  were  established  to  provide  a  direct  lin)c 
with  the  departmental  CFO's.  Secondly,  the  Act  requires 
accountability,  i.e.,  submission  of  detailed  financial  plans 
and  status  reports,  from  the  agencies  and  from  0MB.  Third, 
the  Act  develops  a  strategy  for  producing  audited  financial 
statement  for  each  federal  department.  These  statements  are 
designed  to  professionalize  financial  management  within  the 
government  and  to  emulate  the  practices  in  corporate  America. 

For  DoD,  the  importance  of  this  Act  was  the 
centralization  of  oversight  responsibility  for  financial 
management  training  with  the  DoD  Comptroller. 
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2.  Defense  Management  Report  Decision  985. 


Another  development  in  the  field  of  financial 
management  education  and  training  was  the  initial  Defense 
Management  Report  Decision  (DMRD)  985  issued  for  FY  1991.  The 
primary  issue  that  was  addressed  by  DMRD  985  was,  "What  can  be 
done  to  provide  more,  and  more  effective,  financial  management 
education  and  training  with  greater  efficiency?"  [Ref  5:] 

The  main  result  of  DMRD  985  was  to  charge  the  DoD 
Comptroller  with  the  responsibility  for  ensuring  that  an 
integrated  set  of  high  quality,  cost  effective  financial 
management  and  education  and  training  programs  are  provided 
throughout  DoD.  To  accomplish  this  task,  the  Defense 
Resources  Management  Education  Center  (DRMEC)  located  at  the 
Naval  Postgraduate  School  in  Monterey,  California  was 
assigned  the  responsibility  for  advising  the  DoD  Comptroller 
on  the  progress  of  implementation  of  DMRD  985.  Additionally, 
the  Comptroller  was  given  the  authority  to  designate  a 
Director  for  Financial  Management  Education  and  Training  to 
manage  the  education  and  training  of  this  workforce.  Finally, 
the  Financial  Management  Education  and  Training  Working  Group 
was  estciblished  to  develop  a  management  plan  to  delineate  the 
roles  to  be  played  by  the  DoD  Comptroller  and  the  military 
departments .  [Ref .  5 ; ] 

The  Defense  Management  Report  Decision  985  was  updated 
for  Fiscal  Year  1992.  [Ref.  6:]  The  issue  stated  in  this  DMRD 
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was,  "There  is  a  need  to  provide  more,  and  more  effective, 
financial  management  education  and  training  with  greater 
efficiency."  (Compare  this  definitive  statement  with  the  issue 
raised  by  the  original  DMRD  985  which  asked  "What  can  be  done 
to  provide  more,  and  more  effective,  financial  management 
education  and  training  with  greater  efficiency?")  The  major 
purpose  of  this  revised  DMRD  was  to  implement  recommendations 
developed  as  a  result  of  the  first  DMRD  985.  Recommendations 
included: 

1.  Establish  a  structure  for  the  oversight  of  financial 
management  education  and  training. 

2 .  Provide  career  referral  services  to  DoD  personnel  who  are 
not  presently  provided  those  services;  and 

3.  Establish  a  Resource  Management  Institute. 

The  third  recommendation  provided  that  DRMEC  would 
become  the  Defense  Resource  Management  Institute  (DRMI)  .  DRMI 
was  assigned  several  functions  in  the  area  of  financial 
management.  Also,  DRMI  is  scheduled  to  become  a  Defense 
Support  Activity  which  gives  it  significant  organizational 
status . 

E .  THE  CONCERN  FOR  QUALITY  ASSESSMENT  OF  FINANCIAL  MANAGEMENT 
EDUCATION  AND  TRAINING 

Growing  interest  in  financial  management  and  education  has • 
been  docxunented  in  the  preceding  sections.  The  Chief 
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Financial  Officers  Act  of  1990  and  the  Defense  Management 

Report  Decision  985  highlight  this  concern.  One  outgrowth  of 

this  interest  in  financial  management  and  education  was  the 

concern  by  the  Dod  Comptroller  for  the  "quality"  of  current 

financial  management  education  and  training  programs.  As 

previously  noted,  the  Comptroller  of  DoD  assumed 

responsibility  for  the  oversight  of  financial  management 

programs.  One  direct  result  of  this  was  a  memorand'^m  issued 

by  the  DoD  Comptroller  on  July  3,  1991.  The  subject  of  this 

memorandum  was  "Management  Plan  for  the  Review  of  Financial 

Management  Education  and  Training".  [Ref.  7;]  While  this 

memorandum  addressed  a  broad  spectrum  of  financial  management 

issues,  it  also  dealt  directly  with  the  issue  of  quality 

control .  Under  the  heading  of  Needs  Assessment ,  the  area  of 

Curriculum  Configuration  Management  and  Quality  Control  was 

discussed.  To  quote  from  the  Memorandum: 

Financial  management  course  offerings  fall  generally  into 
two  categories:  core  courses  that  deal  in  basic  concepts 
and  principles  that  are  independent  of  specific 
application  .  .  . ;  and  service  unique  applications  that 
train  personnel  to  perform  relatively  narrow  functions  or 
apply  basic  principles  in  situations  peculiar  to  a 
particular  Service. 

In  the  case  of  core  courses  there  is  a  need  to  ensure  that 
the  course  content  covers  the  subject  matter  at  an 
appropriate  level  of  comprehensiveness  and  sophistication. 

A  second,  related  function  is  that  of  ensuring  that  what 
Is  taught  is  taught  well.  Whether  emalyzlng  new  proposals 
or  evaluating  ongoing  programs  of  Instruction,  assessment 
of  teaching  effectiveness  are  needed.  (Emphasis  added) 

[Ref.  7:] 
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The  specific  task  of  measuring  program  effectiveness  was 
assigned  to  the  Defense  Resources  Management  Education  Center 
(DRMEC) .  [Ref.  7:] 

This  thesis  arose  as  a  part  of  the  effort  of  the  DoD 
Comptroller  and  the  Financial  Management  faculty  at  the  Naval 
Postgraduate  School  to  conduct  research  on  the  quality 
assessment  methods  used  in  DoD  financial  management  education 
and  training  programs . 
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III.  SURVEY  METHODOLOGY 


A.  INTRODUCTION 

A  principal  methodology  employed  in  this  thesis  is  survey 
research.  Surveys  employed  in  empirical  research  typically 
are  random,  that  is,  surveys  are  sent  to  a  randomly  selected 
distribution  of  the  targeted  population  with  the  expectation 
that  only  some  portion  of  the  surveys  will  be  returned.  In 
most  cases  this  approach  is  satisfactory.  However,  one  of  the 
primary  goals  of  this  research  is  to  develop  a  comprehensive 
data  base  of  all  providers  of  financial  management  education 
and  training  within  DoD  and  to  inventory  the  various 
techniques  that  are  employed  by  these  providers  to  assess  the 
quality  of  their  programs.  Therefore,  a  census  rather  than  a 
sample  of  DoD  financial  management  providers  was  conducted. 

One  further  note  on  terminology  is  required.  The  terms 
"survey"  and  "questionnaire"  are  used  in  this  thesis 
interchangeably.  A  questionnaire  is  the  tool  used  in  survey 
research  to  collect  information. 

B.  THE  DATA  BASE  OF  AGENCIES  TO  BE  SURVEYED 

The  actual  survey  that  was  sent  out  to  all  providers  of 
financial  management  education  and  training  programs  is 
included  in  Appendix  B.  The  list  of  DoD  agencies  and 
departments  responding  to  the  survey  is  provided  in  Appendix 
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C.  Agencies  and  departments  which  failed  to  respond  to  the 
survey  is  provided  in  Appendix  D. 

Prior  to  developing  and  mailing  the  survey  to  all 
providers  of  financial  management  education  and  training,  a 
comprehensive  list  of  these  providers  needed  to  be  verified. 
The  principal  sources  for  determining  the  agencies  and 
departments  to  be  surveyed  were: 

(1)  Formal  Schools  Directory.  Third  Edition  [Ref.  8:] 
dated  January,  1992.  This  directory  was  prepared  by  the  DoD 
Training  and  Performance  Data  Center  (TPDC)  which  is  located 
in  Orlando,  Florida.  The  directory  lists  all  schools  and 
training  locations  operated  by  the  Military  Services  and  DoD 
agencies,  including  those  identified  as  financial  management. 
Five  hundred  eighty  nine  training  organizations  are  identified 
in  this  directory. 

(2)  Financial  Management  Data  System  [Ref.  9;]  also 
developed  by  TPDC.  The  Financial  Management  Data  System  is  an 
automated  data  collection  system  which  is  designed  to 
facilitate  the  collection  of  descriptive  and  resource  data  for 
DoD  financial  management  training  and  education  courses.  TPDC 
provided  this  researcher  with  the  latest  edition  of  the  data 
base  which  was  updated  through  January,  1992. 

The  financial  management  data  base  was  cross-referenced  to 
the  Formal  Schools  Directory  [Ref.  8:]  to  ensure  that  the 
latest  available  data  on  providers  of  financial  management 
education  and  training  was  utilized. 
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A  final  check  was  performed  to  ensure  that  all  providers 
of  financial  management  education  and  training  were  included. 
This  included  correspondence  and  communication  with  experts 
both  at  the  Naval  Postgraduate  School  and  with  field  personnel 
in  the  various  military  departments.  The  final  product  or 
listing  of  providers  was  then  prepared. 


C.  DEVELOPMENT  OF  THE  SURVEY  INSTRUMENT 
1.  Research  on  Survey  Methodology 

The  development  of  the  survey  used  to  gather  data  for 
this  research  began  with  an  extensive  review  of  the  literature 
on  survey  design.  [See  references  10  through  21]  One  of  the 
first  issues  that  needed  to  be  addressed  was  what  form  would 
the  survey  questions  take,  e.  g.,  open  or  closed  format, 
scaler,  filtered,  "Open"  questions  are  designed  to  allow  the 
widest  latitude  to  survey  recipients  in  responding  to  the 
survey  question.  An  example  of  an  open  question  would  be:  In 
your  opinion,  what  are  the  most  important  methods  you  employ 
to  ensure  the  quality  of  your  program?  Closed  questions,  on 
the  other  hand,  present  a  listing  of  options  or  preconceived 
responses  for  the  survey  respondence  to  choose  from.  The 
scaler  type  of  question  asks  the  recipient  to  assign  a  "score" 
to  a  particular,  question.  This  score  normally  ranges  from  5 
to  1  if  numerically  weighted  or  from  frequently  to  seldom  in 
verbal  terms.  Filtered  or  filtering  questions  involve  the  use 
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of  "no  opinion"  type  responses  or  simple  "YES/NO"  responses. 
[Ref.  11:  pp.  41-56] 

Since  the  full  spectrum  of  assessment  techniques 
employed  by  DoD  financial  management  education  and  training 
programs  was  not  known,  the  basic  methodological  approach 
en^loyed  in  the  development  of  the  questionnaire  was  the 
"open"  response  format.  This  format  allows  the  recipient  to 
reply  to  the  questions  in  a  manner  best  suited  to  the 
institutional  setting  of  the  respondent.  [Ref.  11:  p.  54] 
Additionally,  filtering  questions  were  developed  to  facilitate 
the  response  to  questions  that  did  not  apply  to  certain 
recipients  but  did  apply  in  general.  Finally,  general  purpose 
or  information  questions  were  included  in  the  question  data 
base. 

A  second  methodological  issue  that  was  addressed  was 
who  should  be  contacted  in  the  survey  process.  Mr.  Peter 
Ewell,  Senior  Associate  with  the  National  Center  for  Higher 
Education  Management  Systems  (NCHEMS)  noted  that  quality 
assessment  of  education  could  be  conducted  in  two  principal 
ways.  These  methods  are: 

(1)  collection  from  the  agencies  and  departments 
themselves  or  an  internal  perspective,  and 

(2)  Collection  of  information  from  the  clients  served  by 
the  agencies  and  departments.  The  clients  are  either  the 
students  themselves  or  the  sponsoring  agencies  which 
subsequently  received  the  students.  This  method  of  data 
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gathering  is  referred  to  as  external,  [Ref.  22]  Due  to  the 
limited  time  frame  and  the  prospect  of  uncertain  response  from 
the  external  sources,  the  internal  method  was  selected. 

The  third  methodological  issue  addressed  was  the  data 
collection  method  to  be  employed  in  the  survey.  The  various 
methods  of  data  collection  are  presented  in  Table  IV  below. 
The  table  lists  the  methods,  a  brief  description  of  the  method 
and  summarizes  the  weaknesses  and  strengths  of  each  method. 
The  information  derived  in  Table  IV  was  derived  from 
Measurement  Errors  in  Surveys  by  Paul  P.  Biemer. 

[Ref.  11;  pp.  237-250] 

Given  the  time  constraints  as  well  as  the  perceived 
length  of  the  questionnaire  on  financial  management  quality 
assessment  techniques,  a  combination  of  telephone  and  mail 
survey  techniques  were  selected.  In  addition  to  the 
probability  of  gaining  higher  reliability  of  the  responses, 
this  method  allowed  the  survey  respondent  to  include 
additionally  ma  erials  to  the  basic  survey  document.  The 
problem  of  non-response,  which  is  normally  a  factor  in  a  mail 
survey,  was  discounted  due  to  the  support  for  the  survey  by 
the  Comptroller  of  DoD.  This  is  discussed  later  in  this 
section. 
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TABLE  IV.  DATA  COLLECTION  METHODS 


Methods  of 

Data 

Collection 

Description 

Advantages 

Disadvantages 

Questionnaire 

Survey  mailed 
to  respondent 

Data  from 
large  number 
of  people. 
Standard  for 
measuring 
change . 

Quick. 

Measure 
attitudes  not 
behavior . 
Requires 
honesty  on 
part  of 
respondent . 

Observation 

Agent  observes 
activity  to  be 
measured . 

Rich 

behavioral 

data. 

Time 

consuming . 
Subjective. 

Telephone 

Interview 

Agent  collects 
survey  data 
via  phone. 

Similar  to 
questionnaire. 

Consistency. 

Time 

consuming . 

Face  to  Face 
Interview 

Survey 

administered 
in  presence  of 
respondent . 

Provides  best 
data. 

Time 

consuming . 
Interview 
bias . 

Source;  Paul  P.  Biemer,  Measurement  Errors  in  Surveys.  1991. 


2.  Initial  Development  of  the  Survey  Questions 

Once  the  decision  had  been  made  by  DoD  Comptroller 
staff  to  perform  a  quality  assessment  of  financial  management 
education  and  training  courses  and  programs,  the  initial 
development  of  the  survey  instrximent  was  undertaken.  In  this 
phase  of  development,  two  steps  were  followed.  The  first  step 
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was  to  review  research  literature  on  survey  methodology. 
(Detailed  in  the  previous  section) . 

The  second  step  was  to  contact  professionals  in  the 
field  of  educational  assessment  to  provide  additional  guidance 
on  the  survey  design. 

One  of  the  more  significant  contributions  at  this 
stage  was  provided  by  Peter  T.  Ewell,  Senior  Associate  at  the 
National  Center  for  Higher  Education  Management  Systems 
located  in  Boulder,  Colorado,  In  his  response  to  a  request 
for  assistance  in  developing  a  comprehensive  survey,  Mr.  Ewell 
recommended  the  following  procedures  be  assessed: 


•  The  nature  of  the  program  with  respect  to  target  audience 
(e.g.,  civilian/military,  rank,  geographical  region)  and 
its  intensity/duration  (e.g.,  one  day,  two-week  on  site, 
one  year  part-time,  correspondence  course) . 

•  The  learning  objectives  of  the  program,  framed  if  possible 
in  terms  of  "expected  outcomes"  of  instruction  i.e., 
particular  elements  of  knowledge  or  skill  that  a  student 
will  exhibit  at  the  progreims  conclusion. 

•  The  particular  methods  or  modes  of  instruction  used  in 
delivering  the  program  (e.g.,  hands-on  training, 
simulation,  classroom  work,  independent  study,  etc.),  and 
in  particular  the  ways  in  which  students'  competency  is 
tested  at  its  conclusion  (e.g.,  paper -and -pencil  exams, 
problems,  rated  demonstrations  of  hands-on  performance, 
etc. )  . 

•  How  the  effectiveness  of  the  program  in  attaining  its 
training  objectives  has  been  evaluated  in  the  past;  if 
possible,  units  should  be  requested  to  attach  copies  of 
any  studies  or  data  on  effectiveness  that  they  routinely 
collect  to  monitor  and  improve  their  own  performance. 
[Ref.  22] 
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Concurrent  with  seeking  outside  guidance,  a  team  of 
Naval  Postgraduate  Financial  Management  professors  provided  a 
series  of  questions  for  inclusion  in  the  survey.  These  were 
in  addition  to  the  questions  developed  as  a  result  of  the 
literature  search  and  contact  with  educational  professionals 
outside  DoD. 

3.  Field  Validation  of  the  Survey  Questions 

Once  the  initial  set  of  survey  questions  had  been 
compiled,  the  "validity"  and  "relevance"  of  these  questions 
needed  to  be  tested.  To  accomplish  this  testing,  selected 
agencies  and  departments  offering  financial  management 
education  and  training  programs  were  contacted  through  a 
telephone  survey.  The  methodology  employed  in  this  telephone 
survey  was  "blind"  response,  i.e.,  the  respondent  was  asked 
general  questions  on  quality  assessment  and  then  asked  what 
questions  they  would  consider  relevant  in  developing  a  survey 
instrument  for  their  facility.  Ten  of  the  31  institutions 
previously  identified  as  providers  of  financial  management 
education  and  training  were  contacted  in  this  process.  The 
providers  were  selected  on  the  basis  of  size,  service  brr -ich 
and  referral  from  previously  contacted  providers.  The  various 
c[uestions,  suggestions  and  pertinent  information  collected 
during  the  telephone  survey  were  added  to  the  data  base  of 
questions  prepared  prior  to  the  telephone  survey.  Appendix  A 
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lists  those  agencies  and  departments  contacted  during  this 
phase  of  survey  preparation. 

4.  The  Initial  Survey  Instrument 

Once  the  agencies  and  departments  were  selected  for 
survey  and  the  survey  questions  to  be  asked  had  been  collected 
and  developed,  the  next  phase  was  to  organize  the  survey 
questions  into  logical  units.  Specifically,  the  questions 
that  had  been  obtained  through  the  research  noted  previously 
had  to  be  organized  into  a  cohesive  survey  document.  The 
organization  of  the  survey  was  viewed  as  essential  for 
con^lete  and  successful  data  collection  on  the  quality 
measures  employed  by  financial  management  education  and 
training  providers. 

As  Paul  Biemer  notes  in  Measurement  Errors  in  Surveys 
"  [Survey]  Questions  are  not  asked  in  isolation  but  are  grouped 
together  in  a  questionnaire."  [Ref.  11:  p.  30]  Further,  "Once 
the  researcher  decides  [what  questions  to  ask] ,  he  needs  to 
determine  the  order  in  which  they  are  presented  to  the 
respondents.  Survey  literature  indicates  that  presentation 
order  strongly  influences  the...  results."  [Ref.  11:  p,  51] 
One  typical  organizational  model  that  may  be  employed 
is  the  "process"  or  "throughput"  model.  This  model 
essentially  has  three  components  -  inputs .  outputs,  and  the 
processing  oZ  inputs  to  outputs. 
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Various  studies  on  evaluation  [Refs.  10,  12,  14,  17, 
18,  19,  20,  21]  have  noted  variations  to  the  simplified 
process  model  described  above.  For  example,  Suchman 
identified  five  types  of  evaluation  that  an  agency  can  employ 
to  measure  its  performance.  He  defined  the  types  as  follows; 


1.  Effort .  Effort  is  equivalent  to  input.  Effort 
evaluation  is  measuring  inputs  as  an  indicator  of  meeting 
the  goals  and  objectives  of  an  organization. 

2.  Effectiveness .  Effectiveness  is  the  measuring  of  the 
outputs  of  an  organization. 

3.  Adequacy .  Adequacy  measures  outputs  in  a  larger 
institutional  setting.  It  measures  outputs  to  needs, 
i.e.,  why  are  the  services  being  provided? 

4.  Efficiency.  Efficiency  attempts  to  relate  outputs  to 
levels  of  inputs. 

5.  Process .  Process  focuses  on  the  process  by  which  inputs 
are  transformed  into  outputs. 

[Ref.  16:  pp.  64-65] 

Stufflebeam  uses  a  similar  dichotomy  to  describe  the 
evaluation  process.  He  divides  evaluation  into  context, 
input,  process,  and  product  categories.  [Ref.  19:  p.  64] 
Stuff lebeam' s  categories  differ  slightly  from  Suchman' s  but 
essentially  view  the  organization  in  a  similar  vein.  The  main 
difference  is  that  the  timing  of  evaluation  is  key  to 
Stuff lebeam' s  model  whereas  Suchman  is  more  concerned  with 
organization  processes. 

Based  on  the  research  conducted  on  organizational  . 
models,  the  process  model  for  organizing  the  survey  instrximent 
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was  adopted  due  to  its  inherent  simplicity.  Questions  were 
segmented  into  the  following  categories  {which  roughly 
correspond  to  various  phases  in  organizations) .  The  first 
three  categories  correspond  to  inputs,  the  fourth  to  process 
and  the  last  to  output.  The  general  categories  and  principle 
questions  or  information  requested  is  listed  below: 


1 .  General  organizational  and  program  information  including: 
respondent,  point  of  contact,  target  audience,  number  of 
financial  management  courses  and  programs  offered,  length 
of  the  progrcim,  numbers  of  times  offered,  annual 
enrollment  and  average  class  size  among  others. 

2.  Program  requirements  and  development:  How  is  the  need 
determined  for  new  courses?  How  is  the  need  tc  update 
or  modify  courses  determined?  Is  course  development 
guided  by  higher  authority?  and  Does  the  agency  or 
department  have  staff  dedicated  to  program  development? 

3.  Instiruction:  What  policies  are  used  to  hire 

faculty/instructors?  What  methods  are  use  to  evaluate 
instruction  and  instructors? 

4.  Ongoing  evaluation:  What  methods  are  used  to  determine 
the  need  for  the  current  program?  What  methods  are  used 
to  determine  student  competency?  Are  formal  methods  used 
for  student  course  evaluation?  and  What  percentage  of 
students  successfully  complete  the  program? 

5.  Progrcim  assessment :  Is  the  performance  of  graduates 

tracked?  What  formal  methods  are  employed  to  solicit 
feedback  from  former  students  and  clients  of  those 
students? 


Once  the  questions  were  organized  as  noted  above,  the 
complete  survey  questionnaire  was  ready  for  initial  pilot 
testing. 
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5.  Pilot  Testing 


Pilot  testing  consisted  of  having  the  survey  document 
reviewed  by  field  personnel  to  determine  if  the  survey  made 
sense,  was  comprehensive  and  addressed  the  primary  areas  of 
financial  management  quality  assessment. 

The  survey  was  given  to  CDR  Glenn  Eberling  who  taught 
the  Navy  Practical  Controllership  Course  at  the  Naval 
Postgraduate  School  in  Monterey,  California  and  to  Mr.  Wade 
Cliendienst  who  was  the  Chief  of  Training  and  Professional 
Development  for  the  US  Army  Audit  Agency  located  in 
Alexandria,  Virginia. 

Both  recipients  reviewed  the  survey  and  offered 
several  additions  to  the  proposed  survey  instrument.  These 
additions  focused  on  the  methods  of  quality  assessment  and  the 
organization  of  the  survey.  Both  pilot  recipients  also 
suggested  clarification  in  the  wording  of  the  questions  and  of 
the  explanatory  paragraphs  included  in  the  survey.  Neither 
Mr.  Cliendienst  nor  CDR  Eberling  are  responsible  for  the 
ultimate  content  of  the  survey.  Their  contribution  to  the 
survey  was  important  and  therefore  merits  recognition. 

6 .  Sponsorship  by  Comptroller  of  the  Department  of 

Defense 

A  final  step  in  developing  the  survey  of  financial 
management  providers  was  to  obtain  the  support  of  the 
Comptroller  of  DoD.  As  noted,  this  thesis  was  the  result  of 
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the  desire  by  the  DoD  Comptroller  and  his  staff  to  assess  the 
quality  of  financial  management  education  and  training 
programs.  Naval  Postgraduate  School  Financial  Management 
Faculty  assisted  in  this  process  through  direct  liaison  with 
the  DoD  Comptroller. 

A  draft  of  the  survey  was  reviewed  by  DoD  Comptroller 
staff  and  a  cover  letter  from  DoD  Comptroller  Sean  O'Keefe  was 
provided  to  introduce  the  survey.  Sponsorship  of  the  survey 
by  the  Comptroller  was  essential  to  ensure  support  from  the 
agencies  and  departments  surveyed.  The  cover  letter  from  the 
DoD  Comptroller  is  included  at  the  end  of  Appendix  B. 
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IV.  ANALYSIS  OF  SXTRVEY  RESPONSES 


A.  INTRODUCTION 

This  chapter  summarizes  the  findings  obtained  from  the 
survey  responses  provided  by  financial  management  education 
and  training  providers.  Appendix  E  provides  a  comprehensive 
review  of  all  survey  responses  from  which  this  chapter  was 
developed. 

Of  the  31  surveys  sent  to  providers  of  financial 
management  education  and  training,  21  were  returned^.  This 
represents  a  response  rate  of  68  percent.  Surveys  returned  by 
the  Center  for  Army  Leadership  and  the  Army's  Judge  Advocate 
General's  School  indicated  that  neither  agency  offered 
financial  management  education  and  training. 

The  response  rate  based  on  a  revised  total  of  29  providers 
(the  original  thirty-one  less  the  two  mentioned  above)  with  19 
responses  is  67  percent^. 


^  The  21  responses  were  received  prior  to  April  30,1992, 
the  cut-off  date  for  inclusion  in  the  research  discussed  in 
this  chapter. 

^  An  effort  was  made  to  obtain  a  100  percent  response 
rate  to  the  survey.  Three  non- responding  agencies  were 
specifically  identified  for  intensive  follow-up  due  to  either 
their  size  or  number  of  course  offerings.  They  were  the  DLA 
Finance  Training  Section,  the  Defense  Systems  Management 
College  and  the  Army  Audit  Agency.  As  a  result  of  these 
efforts,  surveys  from  the  Defense  Systems  Management  College 
and  DLA  Finance  Training  Section  were  received  after  the 
primary  research  on  the  thesis  had  been  completed.  The 
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Another  measurement  of  the  response  is  to  calculate  the 
number  of  financial  management  programs  managed  by  agencies 
and  departments  who  responded  to  the  survey.  By  this 
reckoning,  the  total  number  of  financial  management  programs 
offered  totals  181.  Respondents  to  the  survey  manage  134  of 
the  181  programs.  The  response  rate  based  on  this  measure  is 
79  percent'*. 

Tables  V(A)  and  V{B)  detail  the  survey  questions  asked  of 
respondents  and  the  actual  number  of  responses  provided  to 
each  question.  The  variability  in  responses  to  the  questions 
is  based  on  (l)  the  non -applicability  of  certain  question  to 
the  various  respondents  and  (2)  the  failure  of  certain 
respondents  to  answer  specific  questions.  The  variability  in 
response  rate  effects  the  discussion  which  follows. 


responses  from  these  two  agencies  conformed  to  the  results 
obtained  from  the  nineteen  survey  respondents  noted  in  this 
section.  Also,  Don  Cress  of  the  Army  Audit  Agency  reported 
that  he  intended  to  complete  and  return  the  survey.  A  revised 
response  rate  based  on  22  responses  is  76  percent.  The  other 
non- responding  agencies  were  contacted  if  the  point  of  contact 
was  known,  but  no  other  follow-up  was  conducted. 

*  Including  DLA,  Defense  Systems  Management  College  and 
the  Army  Audit  Agency,  the  response  rate  is  87  percent;  158  of 
181  programs. 
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TABLE  V(A) :  SURVEY  QUESTIONS /NUMBER  OF  RESPONSES 


1  SURVEY  QUESTION 

NO.  OF  RESPONSES 

1. 

TARGET  AUDIENCE 

19 

2. 

PERCENTAGE  OF  CLASS  ATTENDEES 

19 

3  . 

GRADES /RANKS  OF  ATTENDEES 

19 

4. 

NUMBER  OF  COURSES 

19 

5. 

LENGTH 

17 

6. 

TIMES  OFFERED 

15 

7. 

ANNUAL  ENROLLMENT 

17 

8. 

AVERAGE  CLASS  SIZE 

19 

9. 

COLLEGE  CREDITS 

8 

10. 

DEGREE  OR  CERTIFICATE  OFFERED 

19 

11. 

HOW  DETERMINE  NEED  FOR  NEW  COURSES 

18 

12  . 

HOW  DETERMINE  NEED  TO  UPDATE  COURSES 

17 

13. 

SPONSORS 

19 

14. 

COURSE  DEVELOPMENT  GUIDED 

19 

15. 

TYPES  OF  GUIDANCE  RECEIVED 

14 

16. 

HOW  OFTEN  IS  GUIDANCE  RECEIVED 

17 

17. 

ST.\FF  DEDICATED  TO  PROGRAM 

DEVELOPMENT 

18 

18. 

PERCENTAGE  OF  MIL. /CIV.  INSTRUCTORS 

16 

19. 

POLICIES/CRITERIA  TO  HIRE  MILITARY 

11 

20. 

POLICIES/CRITERIA  TO  HIRE  CIVILIANS 

10 

21. 

PROFESSIONAL  QUALIFICATION  STANDARDS 

18 

22. 

CLASSROOM  OBSERVATION 

19 

23  . 

OTHER  METHODS  TO  EVALUATE 

INSTRUCTION 

18 

24  . 

OTHER  METHODS  TO  EVALUATE 

INSTRUCTORS 

18 
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TABLE  V(B 


SURVEY  QUESTIONS 

NO.  OF  RESPONSES 

25. 

PRODUCTIVITY  MEASURES 

19 

26. 

INSTITUTION  ACCREDITED 

19 

27. 

INSTITUTION  REVIEWED 

19 

28. 

NATURE  OF  REVIEW 

9 

29. 

METHODS  OF  INSTRUCTION 

19 

30. 

PERCENTAGE  OF  INSTRUCTION  BY  METHOD 

11 

31. 

TYPES  OF  COURSE  READING  MATERIAL 

16 

32. 

LIBRARY  SUPPORT 

18 

33. 

IMPROVE  LIBRARY  SUPPORT 

6 

34. 

IMPROVEMENTS  PLANNED  IN  LIBRARY 

17 

35. 

COMPUTER  SUPPORT 

19 

36. 

COMPUTER  EXERCISES 

18 

37. 

PROGRAMMATIC  SUPPORT 

19 

38. 

METHODS  TO  DETERMINE  NEED  FOR 

PROGRAM 

18 

39. 

METHODS  TO  ASSESS  VALIDITY, 

RELEVANCE 

19 

40. 

METHODS  TO  DETERMINE  COMPETENCE 

16 

41. 

FORMAL  METHOD  OF  STUDENT  EVALUATION 

19 

42. 

OTHER  METHODS  FOR  STUDENT  EVALUATION 

19 

43. 

PASS  STANDARDIZED  EXAMS  OR  TESTS 

19 

44. 

PERCENTAGE  OF  STUDENTS  COMPLETING 

11 

45. 

GRADUATE  PERFORMANCED  TRACKED 

19 

46. 

FEEDBACK  FROM  FORMER  STUDENTS 

19 

47. 

MOST  IMPORTANT  METHODS  OF  QUALITY 

19 
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B.  SURVEY  RESPONSES 


1.  General  Characteristics  of  Financial  Management 

Education  and  Training  Courses  and  Progreuas 

Of  the  21  responses  received  in  response  to  the 
survey,  19  agencies  provided  detailed  responses  and  two 
agencies  replied  that  they  did  not  conduct  financial 
management  education  and  training. 

The  survey  questionnaire  was  designed  to  gather  basic 
data  on  the  characteristics  of  financial  management  education 
and  training  providers.  Questions  one  through  ten  provided 
the  respondents  with  the  opportunity  to  describe  their  courses 
or  programs  in  terms  of  target  audience;  number  and  length  of 
those  courses  and  programs;  enrollment;  average  class  size; and 
the  level  of  instruction  (  e.  g.,  degree  granting).  Several 
other  questions  also  provided  general  program  information  on 
these  providers.  For  example,  question  32  asked  whether  the 
financial  management  program  was  supported  by  a  library 
containing  extensive  DoD  financial  management  related 
materials?  The  following  material  details  the  general 
characteristics  of  financial  management  education  and  training 
providers  responding  to  the  survey  questionnaire.  Table  VI 
summarizes  the  general  characteristics  of  financial  management 
education  and  training  providers. 

The  respondents  represented  a  wide  spectrum  of 
institutions  providing  financial  management  education  and 
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training.  In  terms  of  military  sponsorship,  five  respondents 
were  components  within  or  sponsored  by  the  Department  of  the 
Army,  three  within  the  Department  of  the  Air  Force,  and  eleven 
by  the  Department  of  the  Navy  (including  one  response  from  the 
Marine  Corps) . 


TABLE  VI !  SUMMARY  OF  GENERAL  CHARACTERISTICS 
OP  FINANCIAL  MANAGEMENT  EDUCATION  AND  TRAINING  PROVIDERS 


GENERAL  CHARACTERISTICS 

NO.  OF  RESPONSES 

MILITARY  SPONSORSHIP: 

ARMY 

5 

AIR  FORCE 

3 

NAVY 

11 

LEVEL  OF  PROGRAM: 

ENTRY  LEVEL 

1 

PROFESSIONAL 

9 

FULL  SPECTRUM 

9 

AVERAGE  ENROLLMENT: 

LESS  THAN  FIFTY 

2 

FIFTY  TO  ONE  HUNDRED 

3 

ONE  HUNDRED  ONE  TO  TWO  HUNDRED 

2 

MORE  THAN  TWO  HUNDRED 

10 

NUMBER  OF  COURSES  OFFERED: 

ONE 

5 

TWO  TO  FIVE 

4 

SIX  TO  TEN 

4 

MORE  THAN  TEN 

6 

TARGET  AUDIENCE: 

MILITARY 

3 

CIVILIAN 

1 

BOTH 

15 

FACULTY  COMPOSITION: 

MILITARY 

4 

CIVILIAN 

4 

BOTH 

11 
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In  addition  to  representing  all  of  the  military 
services,  the  respondents  spanned  the  educational  spectrum 
from  those  providing  basic  military  entry  level  training  to 
those  granting  Master  degrees.  Nine  of  the  nineteen 
respondents  were  classified  as  providing  professional  level 
education  (e.  g.,  serving  higher  level  civilian  and  military 
personnel  or  providing  an  advanced  educational  degree  -  -  See 
question  3.,  Appendix  E.)  .Of  these  nine,  five  offered 
graduate  level  programs  leading  to  the  award  of  a  master's 
degree . 

In  terms  of  the  size  of  the  institutions  responding  to 
the  survey,  average  enrollment  in  programs  or  courses  ranged 
from  a  low  of  36  to  a  high  of  1,670.  The  total  annual  student 
population  as  reported  by  the  respondents  was  7,324.  Another 
measure  of  size  is  the  number  of  courses  offered.  The  range 
of  courses  offered  was  from  one  to  36,  with  the  average  (mean 
and  median)  being  eight. 

The  target  audience  as  reported  by  the  survey 
respondents,  not  surprisingly,  was  both  military  and  civilian 
employees  of  DoD.  Only  one  agency  reported  its  audience  as 
civilian  only,  and  three  responded  military  only.  Fifteen 
agencies  served  both. 

In  terms  of  faculty  composition,  e.  g.,  military, 
civilian  or  a  combination  of  both,  four  agencies  used  civilian 
instructors,  four  military  and  eleven  had  a  mixture  of 
military  and  civilian. 
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The  wide  diversity  in  program  offerings,  size, 
instructors  and  military  sponsor  of  the  respondents  ensured 
that  the  survey  responses  represented  the  entire  spectrum  of 
financial  management  education  and  training  providers. 

2.  Quality  Assurance  Methods  Employed  By  Providers  of 

Financial  Management  Education  and  Training 

The  survey  .  questionnaire  was  designed  to  elicit 
information  on  the  quality  assessment  methods  that  are  used  by 
financial  management  education  and  training  providers.  Since 
the  survey  was  a  self-assessment  of  these  methods,  the 
responses  provided  represent  the  views  or  opinions  of  the 
providers  on  quality  assurance. 

No  attempt  was  made  to  force  the  responses  into  a 
preconceived  model  of  quality  assurance.  Since  the  survey  is 
descriptive,  the  responses  by  providers  of  financial 
management  education  and  training  represent  their 
interpretation  of  cjuality  assurance  and  the  importance  they 
attach  to  the  methods  used  to  assure  quality  in  their  courses 
and  programs. 

The  analytical  framework  employed  in  the  following 
discussion  begins  with  a  detailed  analysis  of  Question  47. 
Question  47  of  the  survey  asks  --  What  are  the  most  Important 
methods  you  employ  to  ensure  the  quality  of  your  program? 
(Emphasis  added) .  The  question  provided  respondents  the 
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opportunity  to  summarize  the  methods  they  used  to  assure 
quality.  The  responses  to  this  question  provide  the  basis  for 
discussing  the  quality  assurance  methods  employed  by  financial 
management  education  and  training  providers. 

The  following  listing  summarizes  those  methods  cited 
by  survey  respondents  to  ensure  the  quality  of  their  programs: 

•  Student  feedback  (11  responses) . 

•  Sponsor  guidance/feedback/support  (6  responses) . 

•  Hiring/ensuring  quality  faculty  (6  responses) . 

•  Feedback/contact  with  clients  (9  responses) . 

•  Tracking  the  post -graduation  performamce  of  students 
(5  responses) . 

•  Monitoring  of  faculty  performance  (3  responses) . 

•  Keeping  abreast  of  changes  in  financial  management 
(4  responses) . 

•  Tracking  student  performance  in  courses  and/or  programs 
(2  responses) . 

The  methods  noted  by  financial  management  education 
and  training  providers  in  their  responses  to  this  survey 
question  (Question  47)  are  supported  by  the  responses  made  to 
other  questions  within  the  survey  questionnaire.  Each  of  the 
areas  cited  above  are  separately  discussed  below  with 
references  to  other  survey  questions  as  appropriate. 
Responses  to  Question  47  are  not  included  inasmuch  as  that 
question  forms  the  basis  for  further  analysis.  [Complete  and 
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detailed  responses  to  the  survey  questions  are  provided  in 
Appendix  E,  and  are  not  specifically  quoted  below] 
a.  Student  Feedback 

The  use  of  student  feedback,  specifically  end- of - 
course  critique  forms,  is  a  response  that  appears  throughout 
the  surveys  returned  by  financial  management  education  and 
training  providers.  In  addition  to  Question  41  which  asks  -- 
Do  you  have  a  formal  method  of  student  course  evaluation?  to 
which  all  nineteen  respondents  affirmatively  replied,  the 
following  questions  and  the  nximber  of  respondents  that  listed 
student  feedback  as  a  prime  determinant  are  as  follows: 


•  Question  11:  How  do  you  determine  the  need  for  new 

courses  or  instruction?  (6  responses) 

•  Question  12:  How  do  you  determine  the  need  to  update  or 
modify  the  instruction  you  currently  offer?  (10 
responses) 

•  Question  23:  Are  other  methods  used  to  evaluate  classroom 
instruction?  [Note:  Question  23  asks  whether  respondents 
use  classroom  observation  to  evaluate  instruction]  (15 
responses) 

•  Question  24:  Are  other  methods  used  to  evaluate 

instructors?  (7  responses) 

•  Question  38:  What  methods  are  used  to  determine  the  need 

for  your  current  progrcim?  [Follow-on  to  questions  ll  and 
12]  (3  responses) 

•  Question  39:  What  methods  are  used  to  determine  the 

validity,  accuracy,  relevance  and  timeliness  of  your 
current  program?  (7  responses) 
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b.  Sponsor  Guidance 


Sponsor  guidance  is  another  common  method  of 
quality  assurance  employed  by  respondents  to  the  survey. 
Question  13  (Do  you  have  one  or  more  sponsors  or  clients  that 
directly  influence  your  program  of  instruction?)  directly 
measures  this  question.  Eighteen  of  the  nineteen  respondents 
acknowledged  sponsor  guidance.  Question  14  (Is  course 
development  guided  or  directed  by  higher  authority?)  also 
addresses  this  area.  Thirteen  of  nineteen  respondents 
responded  "yes"  to  the  question. 

Additionally,  the  following  survey  questions 
applied  to  the  sponsor  guidance;  [Again,  the  number  of 
positive  responses  is  appended  to  the  end  of  each  question] 

•  Question  11:  How  do  you  determine  need  for  new  courses  or 
instruction?  (9  responses) 

•  Question  12:  How  do  you  determine  the  need  to  update  or 
modify  the  instruction  you  currently  offer?  (8  responses) 

•  Question  27:  Is  your  institution  reviewed  by  other 

agencies  [other  than  accreditation]?  (7  responses) 

•  Question  38:  What  methods  are  used  to  determine  the  need 
for  your  current  program?  (6  responses) 

•  Question  39:  What  methods  are  used  to  determine  the 

validity,  accuracy,  relevance  and  timeliness  of  your 
program?  (8  responses) 

c.  Hiring  Quality  Faculty 

Hiring  of  a  quality  faculty  was  the  third  quality 
assurance  method  cited  by  survey  respondents.  In  this  regard, 
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the  issue  of  faculty  "quality"  or  competence  was  addressed 
directly  by  the  survey  in  questions  19  through  25.  (See 
Appendices  A  and  E)  Question  21  is  the  most  specific.  It 
asked  -  Are  professional  qualification  standards  required  for 
faculty/instructors?  Seventeen  of  18  respondents  stated  that 
they  employed  this  method. 

Also  relevant  for  the  discussion  of  quality 
faculty  were  questions  19  and  20.  The  questions  asked  -  What 
policies  and  criteria  are  used  to  hire  military  (Question  19) 
or  civilian  (Question  20)  faculty/instructors  for  your 
program.  Survey  responses  to  both  questions  focused  on 
experience,  education  and  other  factors  that  prepare 
individuals  for  the  teaching  profession.  In  fact  all 
respondents  noted  that  they  used  selected  criteria  to  ensure 
the  hiring  of  qualified  individuals. 

Question  22  through  25  further  addressed  the  issue 
of  instructor  or  instruction  observation  to  promote  quality 
faculty.  However,  the  issue  of  faculty  on-the-job 
performance,  while  relevant  to  a  quality  faculty  is  more 
appropriately  discussed  in  subsection  (f)  Faculty  Performance 
below. 

d.  Feedback  from  Clients 

Client  feedback  was  the  fourth  method  cited  by 
survey  respondents  to  ensure  quality.  Question  13  addressed 
this  method  by  asking  -  -  Do  you  have  one  or  more  sponsors  or 
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clients  that  directly  influence  your  program  of  instruction? 
As  the  question  itself  implies,  the  distinction  made  between 
clients  and  sponsors  was  not  always  made  clear  in  the 
responses  provided  by  respondents.  Only  one  respondent 
specifically  mentioned  a  client  that  influenced  their  program. 
Nevertheless,  other  survey  question  responses  noted  the 
importance  that  financial  management  education  and  training 
providers  gave  to  client  feedback. 

The  following  questions  directly  applied: 

•  Question  11:  How  do  you  determine  the  need  for  new 

courses  or  instruction?  (9  responses) 

•  Question  12:  How  do  you  determine  the  need  to  update  or 
modify  the  instruction  you  currently  offer?  (3  responses) 

•  Question  38;  What  methods  are  used  to  determine  the  need 
for  your  current  program?  (2  responses) 

•  Question  39:  What  methods  are  used  to  determine  the 

validity,  accuracy,  relevance  and  timeliness  of  your 
current  program?  (3  responses) 

Additionally,  client  feedback  is  also  obtained 
through  the  use  of  post -graduation  surveys  to  the  extent  that 
graduates  and  their  supervisors  represent  the  clients  of  the 
courses  and  progreims  offered.  This  area  is  more  fully 
examined  below. 

e.  Post-graduation  Perfonnance  of  Students 

The  post -graduation  performance  of  financial 
management  education  and  training  students  was  the  fifth 
method  of  quality  assurance  indicated  by  survey  respondents. 
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This  method  was  specifically  addressed  in  the  survey  in 
Question  45.  The  question  stated  --  Do  you  track  the 
performance  of  your  graduates?  Seven  of  the  nineteen 
respondents  did  utilize  such  tracking.  Of  these  seven,  six 
used  post -graduation  surveys  for  this  purpose. 
f.  Faculty  Performance 

A  sixth  method  of  quality  assurance  reported  was 
faculty  performance.  This  method  is  similar  to  the  hiring  of 
quality  faculty  previously  discussed  in  subsection  (c)  . 
However,  the  monitoring  of  faculty  performance  is  an  on-going 
process  that,  in  principle,  verifies  the  hiring  policies  and 
procedures  of  financial  management  providers. 

Questions  22  through  25  addressed  the  issue  of 
faculty  performance.  These  questions  with  the  nvimber  of 
affirmative  responses  are  as  follows: 

•  Question  22:  Do  you  use  classroom  observation  to  evaluate 
instruction?  (17  responses) 

•  Question  23:  Are  other  methods  used  to  evaluate  classroom 
instruction?  (15  responses) 

•  Question  24:  Are  other  methods  used  to  evaluate 

instructors?  (12  responses) 

•  Question  25:  Do  you  employ  productivity  measures  to 

evaluate  instructors?  (3  responses) 

One  of  the  prime  measurements  of  faculty 
performance  was  the  use  of  student  critiques  (end- of -course 
evaluations)  and  student  feedback.  For  example,  all  fifteen 


46 


of  the  responses  to  question  23  cited  the  use  of  student 
critiques  as  the  method  used  to  evaluate  classroom 
instruction.  Similarly,  eight  respondents  to  question  24 
noted  student  critiques  as  the  primary  method  employt_d  to 
measure  instructor  performance.  The  method  of  student 
feedback  as  a  quality  assurance  tool  is  discussed  in  detail  in 
subsection  (a)  above. 

g.  Keeping  Abreast  of  Changes 

Keeping  abreast  of  changes  in  financial  nianagement 
was  the  sixth  quality  assurance  method  cited  by  survey 
respondents.  No  specific  survey  question  explicitly  dealt 
with  this  method.  However,  rhe  need  for  financial  management 
providers  to  maintain  currency  of  their  courses  and  programs 
to  ensure  timeliness  and  relevance  is  obvious.  The  need  to 
keep  abreast  of  changes  in  financial  management  was  noted  in 
the  following  survey  cjuestions: 

•  Question  12:  How  do  you  determine  the  need  to  update  or 
modify  the  instruction  you  currently  offer?  (5  responses) 

•  Question  39:  What  methods  are  used  to  determine  the 

validity,  accuracy,  relevance  and  timeliness  of  your 
current  program?  (2  responses) 

h.  Tracking  Student  Performance 

The  final  method  reported  to  ensure  quality  was 
the  tracking  or  monitoring  of  student  performance  while 
attending  the  financial  manac3ment  institution. 
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The  capability  of  an  institution  to  measure  the 
performance  of  its  students  informs  that  institution  on  the 
success  or  failure  of  its  instruction.  The  survey  addresses 
the  issue  of  student  performance  or  competency  in  questions 
40,  43  and  44. 

Question  40  asked  -  -  What  methods  are  employed  to 
determine  student  competence  during  and  upon  program 
completion  (e.  g.,  passing  standardized  tests,  written  course 
work,  observation  of  performance  on  the  job)?  Question  43 
deals  with  the  requirement  for  students  to  pass  national 
examinations.  Only  one  respondent  required  this.  Finally, 
Question  45  ask  respondents  what  percentage  of  students 
entering  the  program  successfully  complete  it?  The  average 
successful  completion  rate  was  in  excess  of  95  percent. 

Additionally,  while  this  method  is  directed  toward 
in-  house  success,  post -graduation  performance  also  informs 
the  institution  on  the  quality  of  their  program.  This  issue 
is  addressed  in  subsection  (e) . 

C.  SUMMARY 

The  preceding  sections  discussed  the  responses  to  the 
survey  of  financial  management  education  and  training 
providers  in  terms  of  (1)  the  number  of  responses;  (2)  the 
general  characteristics  of  financial  management  providers;  and 
(3)  the  methods  that  financial  management  providers  used  to 
ensure  quality.  Table  VII  summarizes  the  methods  used.  The 
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nximber  of  responses  cited  in  the  table  represent  those 
respondents  that  acknowledged  the  use  of  the  quality  method. 
The  percentage  column  is  calculated  on  the  basis  of  the  number 
of  responses  divided  by  the  total  number  of  responses  received 
(i.  e.,  nineteen). 


TABLE  VII.  SUMMARY  OF  QUALITY  ASSURANCE  METHODS 


1  Method 

Responses 

Percentage 

(a) 

Student  Feedback 

11 

57.9 

(b) 

Sponsor  Guidance 

6 

31.6 

(c) 

Hiring  Quality  Faculty 

6 

31.6 

(d) 

Feedback  from  Clients 

9 

47.4 

(e) 

Post -graduation  Performance 

5 

23.3 

(f) 

Faculty  Performance 

3 

15.8 

(g) 

Keeping  Abreast  of  Changes 

4 

21.1 

(h) 

Tracking  Student  Performance 

2 

10.5 
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V.  CONCLUSIONS 


A.  INTRODUCTION 

This  thesis  presented  and  analyzed  the  results  of  a  survey 
designed  to  determine  how  DoD  financial  management  education 
and  training  programs  assess  the  quality  of  their  programs. 
Quality  in  the  context  of  this  thesis  means  providing 
accurate,  valid,  comprehensive  and  up-to-date  information  to 
meet  the  needs  of  clients  and  customers. 

The  thesis  described  the  need  within  DoD  for  financial 
management  education.  It  documented  the  methodology  employed 
in  developing  the  survey.  The  thesis  discussed  the  general 
characteristics  of  financial  management  education  and  training 
programs  and  summarizes  the  methods  employed  to  ensure  the 
quality  of  these  programs. 

B.  ANSWERS  TO  RESEARCH  QUESTIONS 

The  two  research  question  which  guided  this  thesis  were: 

•  What  financial  management  courses  and  programs  are  offered 
within  DoD  and  what  are  the  general  characteristics  of 
these  courses  and  programs? 

•  What  methods  are  employed  to  ensure  the  quality  of 
financial  management  courses  and  programs  within  DoD? 

Chapter  IV  presented  the  findings  of  the  research.  The 
first  research  question  addressed  the  general  characteristics 
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of  financial  management  education  and  training  programs  within 
DoD.  Table  VI  in  Chapter  IV  summarizes  these  characteristics. 

The  respondents  represented  a  wide  spectrum  of 
institutions  providing  financial  management  education  and 
training.  In  terms  of  military  sponsorship,  five  respondents 
were  components  within  or  sponsored  by  the  Department  of  the 
Army,  three  within  the  Department  of  the  Air  Force,  and  eleven 
by  the  Department  of  the  Navy» 

The  respondents  spanned  the  educational  spectrum  from 
those  granting  Master  degrees  to  those  providing  basic 
military  entry  level  training.  In  terms  of  the  size  of  the 
institutions  responding  to  the  survey,  average  enrollment  in 
programs  or  courses  ranged  from  a  low  of  36  to  a  high  of 
1,670.  The  range  of  courses  offered  was  from  one  to  36,  with 
the  average  (mean  and  median)  being  eight. 

The  target  audience  as  reported  by  the  survey  respondents 
was  both  military  and  civilian  employees  of  DoD.  Only  one 
agency  reported  its  audience  as  civilian  only,  and  three 
responded  military  only.  The  remainder  served  both. 

In  terms  of  faculty  composition, e.  g. ,  military,  civilian 
or  a  combination  of  both,  four  agencies  used  civilian 
instructors,  four  military  and  eight  had  a  mixture  of  military 
and  civilian. 

The  second  research  question  addressed  the  methods  of 
quality  assurance  used  by  financial  management  providers.  The 
responses  by  survey  respondents  to  the  question  of  methods 
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used  to  ensure  quality  are  listed  below:  (Each  of  these 
methods  was  extensively  reviewed  and  discussed  in  Chapter  IV, 
pages  39  to  49,  and  siiinmarized  in  TABLE  VII) 

(a)  Student  Feedback 

(b)  Sponsor  Guidance 

(c)  Hiring  Quality  Faculty 

(d)  Feedback  from  Clients 

(e)  Post -graduation  Performance 

(f)  Faculty  Performance 

(g)  Keeping  Abreast  of  Changes 

(h)  Tracking  Student  Performance 

As  noted  in  Chapter  IV,  student  feedback  and  feedback  frcm 
clients  were  the  two  quality  assurance  methods  most  commonly 
cited  by  survey  respondents.  The  survey  results,  however, 
clearly  point  out  the  wide  divergence  of  quality  methods  used 
by  financial  management  providers. 

In  addition  to  the  methods  cited  by  survey  respondents  to 
ensure  quality,  there  are  other  quality  measures  not  mentioned 
in  the  survey  responses.  The  following  two  methods  are 
offered  as  examples  of  other  quality  assurance  tools  that 
could  be  employed  by  financial  management  education  and 
training  providers:  [This  list  is  not  intended  to  be  all 
encompassing;  other  methods  are  certainly  possible.  The 
discussion  follows  the  format  used  in  Chapter  IV,  Section  2] 
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(1)  Accreditation 


The  accreditation  process  provides  an  independent 
check  on  the  performance  of  participating  agencies  and 
departments.  Accreditation  organizations  typically  review 
courses  and  programs  for  content,  instructor  competence,  the 
provision  of  ancillary  services  such  as  libraries,  etc. 

Question  26  asked  -  Is  your  institution  accredited? 
Nine  of  the  nineteen  respondents  replied  affirmatively.  The 
accrediting  organizations  included  the  North  Central 
Association  of  Colleges  and  Universities,  Commission  on 
Institutions  of  Higher  Management,  the  American  Council  on 
Education  and  the  Western  Association  of  Schools  and  Colleges. 

Obviously,  the  accreditation  process  is  not  applicable 
to  all  of  the  respondents  to  the  survey.  However,  for  those 
that  it  does  apply,  the  accreditation  process  appears  to  be 
designed  to  ensure  the  quality  of  course  and  program 
offerings . 

(2)  Adequate  Institutional  Support 

Another  possible  method  for  ensuring  quality  is  to 
have  adequate  institutional  support  for  the  educational  course 
or  program. 

The  survey  questionnaire  asked  several  generic 
questions  on  institutional  support.  Those  questions  were: 

•  Question  32:  Is  your  institution  supported  by  a  library 
containing  extensive  DoD  financial  management  related 
materials? 
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•  Question  35:  Is  your  program  supported  by  a  computer 
center  or  laboratory? 

•  Question  37:  Is  your  program  well  supported  with  respect 
to  administrative  staff,  funding  for  guest  speakers,  etc.? 

Survey  respondents  had  mixed  responses  to  these 
question.  For  example,  50  percent  of  respondents  reported 
that  their  program  was  not  supported  by  a  library  with 
extensive  DoD  material.  Similarly,  seven  of  nineteen 
respondents  answered  that  their  program  was  not  well  supported 
with  respect  to  administrative  staff,  funding  for  guest 
speakers,  etc.  [However,  those  providing  detailed  answers  (4 
responses)  focused  on  funding  for  guest  speakers  as  the  only 
deficiency  in  this  area]  Nevertheless,  it  seems  clear  that 
adequate  support  is  a  measure  of  quality. 

While  it  may  seem  obvious  that  developing  and 
maintaining  quality  programs  necessitates  the  involvement  not 
only  of  the  provider  but  also  of  the  sponsor  and  clients 
(i.e.,  the  students  and  the  "recipients"  of  these  students), 
the  overall  response  rate  indicates  that  this  is  not  the  case. 
If  there  is  a  concern  for  quality,  the  imperative  to 
understand  and  meet  the  needs  of  the  client  is  obvious. 

Furthermore,  the  divergence  of  responses  suggest  that 
more  effort  on  the  part  of  DoD  needs  to  be  expended  toward  the 
development  of  quality  assurance  methods  that  ensure  financial 
management  education  and  training  is  of  the  highest  quality 
and  is  relevant,  timely  and  comprehensive. 
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C.  AREAS  FOR  FURTHER  RESEARCH 


In  researching  the  quality  assurance  methods  employed  by 
providers  of  financial  management  and  education,  several  areas 
of  further  research  were  noted.  These  include: 

•  Can  a  model  of  quality  assurance  be  developed  and  applied 
to  financial  management  education  and  training  programs? 

•  Can  the  various  quality  assurance  methods  cited  by 
financial  management  education  and  training  providers  be 
employed  by  all  such  providers? 

•  Are  certain  quality  methods  used  by  financial  education 
and  training  programs  more  effective  than  others;  and  if 
so  why? 

One  final  area  for  further  research  is  the  effort  to 
obtain  a  100  percent  survey  response  in  order  to  develop  a 
complete  and  comprehensive  data  base  on  general 
characteristics  and  quality  methods  employed  by  financial 
management  education  and  training  programs. 
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APPENDIX  A 


DoD  FINANCIAL  MANAGEMENT  SCHOOLS  PERSONNEL 
CONTACTED  IN  SURVEY  DEVELOPMENT 

1.  Mr.  William  Pease 

US  Army  Finance  School 
Fort  Benjamin  Harrison,  IN 

2 .  Mr .  Paul  Wagner 

US  Army  Management  Engineering  College 
Rock  Island  Arsenal,  IL 

3.  Mr.  Wade  Cliendinst 
US  Army  Audit  Agency 
A1 exandr ia ,  VA 

4.  Mr.  Jack  Me  Murchy 

US  Army  Logistics  Management  College 
Port  Lee,  VA 

5 .  Mr .  Robinson 

3750TH  Technical  Training  Group 
Sheppard  Air  Force  Base,  TX 

6 .  Dr .  Richard  Lestor 
Air  University 

Professional  Military  Comptroller  School 
Maxwell  Air  Force  Base,  AL 

7 .  Lieutenant  Colonel  Stephen  Adams 
Air  University 

Professional  Military  Comptroller  School 
Maxwell  Air  Force  Base,  AL 

8.  Commander  Glenn  Eberling 
Naval  Postgraduate  School 
Professional  Comptroller  Course 
Monterey,  CA 

9.  Ms.  Nina  Allen 
American  University 
Washington,  DC 

10.  Dr.  C.  W.  Hoofnagle 
Fleet  Tra:ining  Center 
Norfolk,  VA 
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APPENDIX  Bt 


DEPARTMENT  OF  DEFENSE  FINANCIAL  MANAGEMENT  EDUCATION 

AMD  TRAINING  QUALITY  ASSESSMENT  SURVEY 

PURPOSE :  This  survey  questionnaire  is  designed  to  determine  how 
DoD  financial  management  education  and  training  programs  assess  the 
quality  of  their  programs.  Quality  in  the  context  of  this  survey 
means  providing  accurate,  valid,  comprehensive  and  up-to-date 
information  to  meet  the  needs  of  your  customers  or  clients. 
Quality  begins  with  the  determination  of  the  need  for  the  program 
and  continues  through  the  monitoring  of  the  performance  of 
graduates.  Quality  consists  of  providing  the  correct  instruction 
for  the  appropriate  target  population. 

The  following  series  of  questions  is  designed  to  allow  you  to 
provide  information  on  quality  assessment  techniques  employed  by 
your  organization.  We  encourage  you  to  answer  the  questions 
completely  and  add  additional  information  wherever  appropriate. 
Please  attach  additional  pages  and  materials  as  necessary  to 
provide  complete  answers. 


ORGANIZATIONAL  INFORMATION; 

Respondent '  s  Ncune :  _ 

Position;  _ 

Agency/ Con^onent  Name :  _ 

Address : 


Phone  Number: 
(DSN/Autovon) 
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Please  provide  the  title  or  name  of  the  instrv  ::tional  program 
offered  at  your  institution: 


GENERAL  PROGRAM  INFORMATION: 

1.  What  is  the  target  audience  for  your  program? 

(CIVILIAN)  (MILITARY)  (BOTH) 

2 .  What  are  the  approximate  percentages  of  class  attendees? 

CIVILIAN: _  % 

MILITARY: _  % 

3.  What  are  the  ranges  of  grades  or  ranks  of  your  attendees? 

CIVILIAN : _ 

(GS  or  GM  -  to  GS  or  GM  -  ;  SES  Level) 

MILITARY : _ 

4.  How  many  separate  courses  in  financial  management  are 
offered  within  your  program? 


5 .  What  is  the  length  of  your  program  in  days ,  weeks  or 
months?  _ 

6  .  How  many  times  per  year  is  it  offered? _ 

7.  What  is  the  annual  enrollment  in  your  program? 


8.  What  is  the  average  (mean)  class  size? 
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9. 


If  your  program  provides  course  credit  for  college,  how 
many  credits  are  offered  in  the  total  program? 


10.  Do  students  completing  the  program  receive  a  degree  or 
certificate? 

(YES)  (NO) 

If  yes,  what  is  the  title  of  this  award? 


The  rest  of  the  questionnaire  is  organized  into  the  following 
parts:  (1)  Program  requirements  and  development  (2)  Instruction  (3) 
Ongoing  evaluation  and  (4)  Program  assessment.  Some  of  the 
questions  require  a  simple  YES/NO  response,  other  aslc  for  short 
responses,  helpful  comments  and  recommendations.  We  aslc  you  to 
carefully  complete  this  survey  so  that  we  can  develop  a 
comprehensive  data  base  of  quality  assessment  measures  employed  in 
financial  management  education  and  training. 


PROGRAM  REQUIREMENTS  AND  DEVELOPMENT: 

11.  How  do  you  determine  need  for  new  courses  or  instruction? 


12.  How  do  you  determine  the  need  to  update  or  modify  the 
instruction  you  currently  offer? 
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13 .  Do  you  have  one  or  more  sponsors  or  clients  that  directly 
influence  your  program  of  instruction? 

(YES)  (NO) 

If  yes,  who  is/are  the  sponsor (s)? 


14. 

Is  course  development  guided  or  directed  by  higher 
authority? 

(YES)  (NO) 

If  so,  who? 

15. 

What  types  of  guidance  do  you  receive? 

16. 

How  often  do  you  receive  such  guidance? 

17. 

Do  you  have  staff  dedicated  to  program  development? 

(YES)  (NO) 

If  yes,  how  many? 

INSTRUCTION; 

18.  What  are  the  number  of  and  percentages  of  civilian  and/or 
military  instructors  in  your  program? 

CIVILIAN:  Number _ Percentage _ 

MILITARY :  Niomber _ Percentage _ 
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19. 


What  policies  and  criteria  are  used  to  assign  or  hire 
military  faculty/instructors  for  your  program?  Please 
attrch  any  material  or  guidance  that  describes  these 
policies  or  criteria  to  your  survey  response. 


20.  What  policies  and  criteria  are  used  to  appoint  or  hire 
civilian  faculty/instructors  for  your  program?  Please 
attach  any  material  or  guidance  that  describes  these 
policies  and  criteria  to  your  survey  response. 


21.  Are  professional  qualifications  standards  required  for 
faculty/ instructors  (e.g.,  degrees,  prior  experience, 
completion  of  an  instructor  training  program) ? 

(YES)  (NO) 

If  yes,  please  describe.  Attach  written  guidance  or  material 
that  describes  such  qualifications  to  your  survey  response. 


22 .  Do  you  use  classroom  observation  to  evaluate  instruction? 
(YES)  (NO) 

If  yes,  who  performs  this  observation? 


How  often  is  it  performed? 


61 


23. 


Are  other  methods  used  to  evaluate  classroom  instruction? 


(YES)  (NO) 

If  yes,  please  describe: 


24.  Are  other  methods  used  to  evaluate  instructors? 
(YES)  (NO) 

If  yes,  please  describe: 


25.  Do  you  employ  productivity  measures  to  evaluate 
instructors? 

(YES)  (NO) 

If  so,  please  describe.  Attach  any  materials  that 
describe  this  process  to  your  survey  response. 


26.  Is  your  institution  accredited? 

(YES)  (NO) 

If  yes,  by  whom?  _ 

How  often  is  the  accreditation  review  performed? 
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27. 


Is  your  institution  reviewed  by  other  agencies? 


(YES)  (NO) 

If  yes,  by  whom  and  how  often? 


28.  What  is  the  nature  and  extent  of  this  review?  Please 

attach  any  materials  that  describes  this  process  to  your 
survey  response. 


29.  What  methods  or  modes  of  instruction  are  utilized  in  your 
program  (e.g.,  lecture,  discussion,  simulation, 
independent  study) ? 


30.  What  percentage  of  instruction  is  delivered  in  each  mode 
you  identified? 


31.  What  types  of  course  reading  materials  are  used  in  your 

program  (e,g.,  textbooks,  published  articles,  DoD  official 
documents,  self -generated  materials)? 
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32. 


Is  your  program  supported  by  a  library  containing 
extensive  DoD  financial  management  related  materials? 


(YES)  (NO) 

33.  How  would  you  like  to  improve  library  support  for  your 
program? 


34.  Are  there  improvements  in  your  library  services  planned? 
(YES)  (NO) 

If  yes,  please  explain  these  improvements  and  indicate 
when  they  will  occur. 


35.  Is  your  program  supported  by  a  computer  center  or 
laboratory? 

(YES)  (NO) 

If  no,  what  computer  resources  are  employed  at  your 
institution? 


36.  Does  your  instructional  program  include  computer  exercises 
or  assignments? 

(YES)  (NO) 

If  yes,  please  describe  how  computers  are  utilized. 
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37.  Is  your  program  well  supported  with  respect  to 

administrative  staff,  funding  for  guest  speakers,  etc.? 

(YES)  (NO) 

What  improvements  in  support  would  be  desirable? 


38.  What  methods  are  used  to  determine  the  need  for  your 


39.  What  methods  are  used  to  determine  the  validity,  accuracy, 
relevance  and  timeliness  of  your  current  program? 


40.  What  methods  are  employed  to  determine  student  competence 
during  and  upon  program  completion  (e.g.,  passing 
standardized  tests,  written  course  work,  observation  of 
performance  on  the  job)? 
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41. 


Do  you  have  a  formal  method  of  student  course  evaluation? 


(YES)  (NO) 

If  yes,  describe  this  process  and  provide  the  form  used  to 
gather  student  evaluations  with  your  response. 


42 .  Are  other  methods  used  to  permit  students  to  evaluate  your 
instructional  program? 

(YES)  (NO) 

If  yes,  describe  these  methods: 


43 .  Are  your  students  required  to  pass  any  standardized  DoD  or 
national  examinations  (e.g.,  CPA,  CMA) ? 

(YES)  (NO) 

If  yes,  please  identify: _ 

44 .  What  percentage  of  students  entering  your  program 

successfully  complete  it?  _ % 

PROGRAM  ASSESSMENT: 

45.  Do  you  track  the  performance  of  your  graduates? 

(YES)  (NO) 

If  yes,  how  is  this  done? 
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46.  Are  formal  methods  employed  to  solicit  feedback  from 
former  students  and  clients  about  the  utility  of  your 
program? 

(YES)  (NO) 

If  yes,  please  explain  how  this  is  done  and  provide 
examples  of  such  feedback. 


47.  In  your  opinion,  what  are  the  most  important  methods  you 
employ  to  ensure  the  quality  of  your  program? 


48.  Is  there  any  other  information  that  you  would  offer  on 
quality  assessment  at  your  institution  that  we  did  not 
request?  Please  provide  comments  as  appropriate. 
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COMPTROLLER  OF  THE  DEPARTMENT  OF  DEFENSE 
WASHINGTON.  DC  20301-1  100 


FEB  2  7  1992 


MEMORANDUM  FOR  RECIPIENT  OF  THE  FINANCIAL  MANAGEMENT  QUALITY 

ASSESSMENT  SURVEY 


SUBJECT:  Financial  Management  Quality  Assessment  Survey 


Our  education  and  training  community  has  been  busy  the  past 
year  analyzing  how  financial  management  instruction  is  provided 
throughout  the  Department.  As  part  of  this  effort  I  have  asked 
that  a  survey  of  quality  assessment  practices  be  undertaken  to 
provide  a  baseline  from  which  we  can  decide  how  to  direct  future 
initiatives  for  improving  financial  management  education  and 
training. 

The  survey  attached  is  intended  to  gather  information  on 
how  financial  management  education  and  training  institutions 
currently  assure  that  the  instruction  they  provide  is  of  high 
quality  and  is  as  up-to-date  as  possible,  I  ask  that  you  answer 
the  survey  questionnaire  carefully  and  completely,  and  return  it 
by  the  deadline  indicated.  I  can  assure  you  that  this 
information  will  be  used  in  determining  how  financial  management 
education  and  training  will  be  delivered  in  the  future.  Your 
contribution  to  the  process  of  strengthening  financial 
management  instruction  is  appreciated. 


Sean  O' Keefe f 

Attachment 
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APPENDIX  C 


DoD  FINANCIAL  MANAGEMENT  AGENCIES  AND  DEPARTMENTS 
RESPONDING  TO  THE  SURVEY 

ARMY  FINANCIAL  MANAGEMENT  SCHOOLS: 


US  Army  Finance  School 

Fort  Benjamin  Harrison,  IN  46216-5640 

US  Army  Management  Engineering  College 
Rock  Island  Arsenal,  IL  61229-7040 

Syracuse  University 
Army  Programs  Office 
310  School  of  Management 
Syracuse,  NY  13244-2130 

US  Army  Engineer  Division 
Directorate  of  CE  Training  Management 
Huntsville,  AL  35807-4301 

US  Army  Logistics  Management  College 
Fort  Lee,  VA  23801-6056 

*  The  Judge  Advocate  General's  School 
Contract  Law  Division 
Charlottesville,  VA  22903-1781 

*  Center  for  Army  Leadership 
Fort  Leavenworth,  KS  66027 


AIR  FORCE  FINANCIAL  MANAGEMENT  SCHOOLS: 


3750TH  Technical  Training  Group 
Sheppard  Air  Force  Base;  TX  76311-5434 

Air  Force  Institute  of  Technology 

Wright -Patterson  Air  Force  Base,  OH  45433-6583 
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Air  University 

Professional  Military  Comptroller  School 
Mcucwell  Air  Force  Base,  AL  36112-5712 


NAVY  FINANCIAL  MANAGEMENT  SCHOOLS: 


Naval  Postgraduate  School 
Department  of  Administrative  Sciences 
Monterey,  CA.  93943-5022 

American  University 
Washington,  DC  20016-8070 

Troy  State  University 

P.  O.  Box  1032 

Fort  Meyer,  VA  22211 

Navy  Comptroller 

Program  Mcmagement  Office 

Naval  Air  Station  Pensacola 

Building  62 5D 

Pensacola,  FL  32508-5175 

Fleet  Training  Center 
Norfolk,  VA  23511 

Civil  Engineer  Corps  Officers  School 
Port  Hueneme,  CA  93043-5002 

Naval  School  of  Health  Sciences 
Bethesda,  MD  20814-5033 

Fleet  Aviation  Specialized  Operational  Training  Group 
Norfolk,  VA  23511 

Navy  School  of  Manpower  Management 
Norfolk,  VA  23511 

Naval  Military  Personnel  Command 
Navy  MWR  Recreation  Training  Unit 
Patuxent  River,  MD  20670 
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MARINE  CORPS  FINANCIAL  MANAGEMENT  SCHOOLS: 


Financial  Management  School 
Marine  Corps  Service  Support  Schools 
Camp  Lejeune,  NC  28542-5050 


NOTE:  An  asterisks  preceding  the  name  indicates  that  a 

response  was  received.  However,  the  response  was  that  the 
agency  or  department  did  not  perform  financial  management 
education  and  training. 
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APPENDIX  D: 


DoD  FINANCIAL  MANAGEMENT  AGENCIES  AND  DEPARTMENTS 
WHICH  FAILED  TO  RESPOND  TO  THE  SURVEY 

ARMY  FINANCIAL  MANAGEMENT  SCHOOLS: 


US  Army  Audit  Agency 
3101  Park  Center  Drive 
Alexandria,  VA  22302-1596 

US  Army  National  Guard  Bureau 
Washington,  DC  20310-2500 

Army  Reserves  Readiness  Training  Center 
Fort  McCoy,  WI  54656-5000 

Office  of  the  Deputy  Chief  of  Staff  for  Personnel 
The  Pentagon 

Washington,  DC  20310-0300 

Defense  Systems  Management  College 
Business  Management  Department 
Fort  Belvoir,  VA  22060-5426 


AIR  FORCE  FINANCIAL  MANAGEMENT  SCHOOLS; 


3400TH  Technical  Training  Group 
Keesler  Air  Force  Base,  MS  39534-5000 


NAVY  FINANCIAL  MANAGEMENT  SCHOOLS: 


Service  School  Command 
San  Diego,  CA  92133-3000 

Naval  Technical  Training  Center 
Meridian,  MS  39309-5200 
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Department  of  the  Navy 
Progrcun  Information  Center 
The  Pentagon 
Washington,  DC  20350 


DEFENSE  LOGISTICS  AGENCY  SCHOOL; 


DLA  Finance  Center  Training  Section 
DCPSO-PF  Building  150 
P.  O.  Box  3990 
Columbus,  OH  43213-5000 


Note:  The  DLA  Finance  Training  Center  and  the  Defense 
Systems  Management  College  responded  to  the  survey  after  the 
primary  research  was  con^lete.  The  Army  Audit  Agency  is  in 
the  process  of  completing  the  survey.  Their  responses  are 
not  included  in  the  analysis  presented  in  Chapter  IV  or  the 
detailed  responses  provided  in  Appendix  E. 
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APPENDIX  E: 


DETAILED  SURVEY  RESPONSES 

This  appendix  presents  the  responses  to  the  survey 
questions.  Appendix  B  is  the  actual  survey  instrument  that 
was  sent  to  all  financial  management  education  and  training 
providers.  See  Appendix  C  for  the  listing  of  these  agencies 
and  departments  which  responded  to  the  survey.  Appendix  D 
list  those  providers  who  did  not  respond  to  the  survey. 
Chapter  IV  of  this  thesis  summarizes  the  findings  resulting 
from  the  survey. 

This  appendix  is  organized  by  question.  Those  questions 
that  required  a  "YES"  or  "NO"  response  are  indicated  by  an 
asterisk  preceding  the  question  number. 

Each  question  is  identified  and  followed  by  a  summary  of 
the  responses.  Where  appropriate,  comments  provided  by  survey 
respondents  are  included. 
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1.  What  la  the  target  audience  for  vour  pr^g-ram? 


The  response  option  for  this  question  was  civilian, 


military  or  both. 


RESPONSE 

NUMBER 

PERCENTAGE 

CIVILIAN 

1 

5.3  % 

MILITARY 

3 

15.8  % 

BOTH 

15 

78.9  % 

TOTAL  RESPONSES 

19 

100  % 

While  the  vast  majority  of  respondents  replied  "BOTH",  the 
majority  of  students  are  military.  For  example,  several 
agencies  replying  "BOTH"  reported  military  percentages  in  the 
90  percentile  range.  The  sole  agency  reporting  only 
"CIVILIAN"  was  American  University.  American  University 
offers  a  graduate  level  program  for  civilian  employees  of  the 
Department  of  the  Navy. 

2 .  What  are  the  aopm^ltnate  percentages  of  class 
attendees? 

The  responses  to  this  question  varied,  but  as  noted  above 
the  majority  of  students  are  military.  Based  on  the  19 
surveys  received,  approximately  80  %  of  all  reported  students 
are  military  members. 
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3. 


What  are  the  ranges  of  grades  or  ranks  of  vour 
attendees? 


GRADE/RANK 

29TJKBER 

PERCENTAGE 

ENTRY  LEVEL 

1 

5  *  2 

"PROFESSIONAL" 

9 

47.4 

FULL  SPECTRUM 

9 

47.4 

TOTAL 

19 

100.0 

Of  the  19  responses  received,  nine  provide  financial 
management  education  for  "professionals".  Professional  in 
this  analysis  was  defined  as  GS/GM  7's  and  above  or  military 
officers.  Only  one  respondent  provided  services  for  junior 
enlisted  or  civilian  personnel.  The  balance  of  providers 
offered  financial  management  education  across  the  full 
spectrum  of  civilian  and  military  pay  grades. 

4 .  How  many  separate  courses  In  financial  management  are 
offered  within  vour  oT-oty-ram? 


COURSES 

NUMBER 

PERCENTAGE 

1 

5 

26.3 

2-5 

4 

21.1 

5-10 

4 

21.1 

>  10 

6 

31.5 
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The  range  of  courses  offered  as  reported  by  survey 
respondents  was  from  one  to  36.  Detailed  reading  of  the 
responses  indicate  that  providers  of  financial  management 
education  and  training  have  difficulty  defining  courses.  This 
particular  question  was  designed  to  identify  the  number  of 
separate  courses  vice  the  number  of  programs  offered.  It  is 
not  clear  from  the  available  survey  data  that  consistency  in 
course  definition  was  employed  by  the  respondents. 

For  example,  based  on  the  survey  responses,  the  total 
number  of  courses  reported  was  163.  Based  on  information 
previously  collected  this  number  was  expected  to  be  181. 
Further  analysis  is  required  to  understand  the  cause  of  this 
discrepancy.  The  most  likely  cause  of  the  discrepancy  is  the 
difficulty  agencies  have  in  identifying  or  categorizing 
courses  as  "financial  management". 

5 .  What  is  the  length  of  vour  p-rncp-ram  <n  t^ays.  weeks  or 
months? 


LENGTH 

NUMBER 

PERCENTAGE 

5  DAYS  OR  LESS 

4 

23.5 

ONE  MONTH 

2 

11.8 

1  MONTH  -  1  YEAR 

4 

23.5 

ONE  YEAR  OR  MORE 

7 

41.2 

TOTAL  RESPONSES 

17 

100.0 

77 


This  question  again  points  out  the  difficulty  in  the 
distinction  between  "courses"  and  "programs".  While  in  some 
cases  they  may  be  the  same,  e.  g. ,  the  course  and  program 
constitute  the  only  offering  at  an  agency  or  department,  the 
majority  of  survey  respondents  replied  on  the  length  of 
courses  vice  programs. 


6 .  Hnw  wuinv  ttmea  Per  vear  Is  It  offered? 


FREQUENCY 

NUMSER 

PERCENTAGE 

ONCE 

3 

20.0 

TWO  TO  FIVE 

6 

40.0 

SIX  TO  TEN 

2 

13.3 

MORE  THAN  TEN 

4 

26.7 

TOTAL  RESPONSES 

15 

100.0 

7 .  What  Is  annual  enrollment  In  vour  orngi-Mm? 


ENROLLMENT 

NUMBER 

PERCENTAGE 

LESS  THAN  50 

2 

11.8 

50  TO  100 

3 

17.6 

101  TO  200 

2 

11.8 

MORE  THAN  200 

10 

58.8 

TOTAL  RESPONSES 

17 

100.0 
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The  average  (mean  )  enrollment  based  on  the  seventeen 
responses  was  431  students.  The  range  of  students  was  from  36 
to  1,670.  The  annual  total  student  population  reported  by  the 
responding  agencies  totalled  7,324. 


8 .  What  la  the  average  class  size? 


SIZE 

NUMBER 

PERCENTAGE 

TEN  OR  LESS 

1 

5.3 

TEN  TO  TWENTY 

7 

36.8 

TWENTY  TO  THIRTY 

9 

47.4 

MORE  THAN  THIRTY 

2 

10.5 

TOTAL  RESPONSES 

19 

100.0 

The  average  (mean)  class  size  as  reported  by  respondents 
was  23.4.  The  range  was  from  9  to  60  students. 


9 .  If  vour  p-roq-r^m  provides  course  credit  for  college, 

how  many  credits  are  offered  In  the  total  pT-rtefram? 


COLLEGE  CREDIT  GRANTED 

NUMBER 

PERCENT 

MASTER  DEGREE 

5 

26.3 

CREDITS  ONLY 

3 

15.8 

NO  COLLEGE  CREDIT 

11 

57.9 

TOTAL  RESPONSES 

19 

100.0 
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Eight  responses  were  received  on  this  question.  Of  the 
eight  agencies  responding,  five  were  degree  granting  at  the 
Master  level  with  graduate  credits  ranging  from  42  to  85. 
Three  other  agencies  offered  college  credits  of  6  hours,  16 
hours,  and  from  3  to  16  hours  respectively. 

*10.  Do  ffrmtpletlna  the  t>-rr>g-ram  receive  a  degree  or 
certificate? 


RESPONSE 

PERCENTAGE 

YES 

17 

89.5 

NO 

2 

10.5 

TOTAL  RESPONSES 

19 

100.0 

If  ves.  what  Is  the  title  of  this  award? 

Of  the  19  agencies  responding  to  this  question,  ten  issued 
certificates  of  completion,  two  issued  certificates  of 
training,  one  issued  a  diploma  and  five  issued  masters 
degrees.  Two  agencies  do  not  provide  a  degree  or  certificate 
to  graduates.  The  total  exceeds  19  due  to  one  multiple 
response. 
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11.  How  do  you  determine  need  for  new  courses  or 

Instruction? 


The  following  comments  were  provided: 


•  Coordination  with  the  Defense  Finance  and  Accounting 
Service;  Coordination  with  the  Comptroller  of  the  Army  or 
Assistant  Secretary  of  the  Army  for  Financial  Management; 
Coordination  with  the  Army's  Training  and  Doctrine 
Command;  and  comments  from  the  field. 

•  Changes  in  the  operating  environment  in  DoD. 

•  Feedback  from  participants;  initiatives  by  faculty  and 
directors  of  programs;  guidance  from  the  Assistant 
Secretary  of  the  Army  for  Financial  Management. 

•  Assigned  course  proponents  assess  preliminary  needs  prior 
to  an  annual  market  survey  which  determines  definitive 
needs . 

•  New  course  start-up  requirements  are  detailed  in  Army 
regulation  350-3  and  are  in  the  updated  Training  and 
Doctrine  Command's  Training  Requirements  Analysis  System. 

•  Inputs  by  users  and  field  activities. 

•  New  course  requirements  are  identified  by  the  customers, 
usually  through  the  use  of  Air  Force  form  19  -  "Request  to 
Establish  New  Continuing  Education  Short  Course". 
Biannual  curriculum  reviews  with  DoD  customers,  and 
proceeding  of  functional  boards  can  also  identify  needs. 

•  Observation  of  changes  in  governmental  and  academic 
practices  and  problems.  Consultation  with  the  program 
sponsor. 

•  Communication  with  the  Navy's  contract  officer  and  adjunct 
professors  who  are  professionals  in  the  field. 

•  New  course  and  instructional  needs  are  determined  by  on¬ 
going  feedback  from  students,  alumni,  faculty  and  our 
educational  service  officers. 
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•  Needs  survey  of  major  claimants,  the  career  planning  board 
and  participation  in  financial  management  professional 
associations . 

•  Fleet  inputs. 

•  Student  feedback.  Also  input  from  the  sponsor  of  the 
course . 

•  Input  from  the  claimant;  review  of  changes  to  official 
directives;  input  from  field  activities;  input  from  class 
graduates;  input  from  DoD;  review  of  civilian  sector 
approaches . 

•  Fleet  needs;  taskings  from  higher  authority;  critiques 
from  other  courses. 

•  Program  changes;  student  and  claimant  feedback. 

•  Validated  demands  from  field  organizations  for  skills 
training. 

•  The  normal  cycle  for  determination  of  a  new  course  of 
instruction  begins  with  a  change  to  operation  in  the 
community.  The  school  will  get  input  from  headquarters 
that  a  task  or  course  needs  instruction.  Initially,  we 
get  input  from  post -graduation  surveys  which  indicate  that 
new  instruction  is  needed.  Frequently,  we  distribute 
field  surveys  for  more  inputs. 


12 .  How  do  you  determine  the  need  to  update  or  modify  the 

Instruction  vou  currently  offer? 


The  following  comments  were  provided; 


•  Changes  in  regulations  or  new  directives;  changes  in 
computer  programs  related  to  financial  management;  lessons 
learned,  e.  g./  Desert  Storm;  results  of  enlisted  and 
officer  development  tests. 
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•  Changes  in  regulations,  directives  or  operating 
environment . 

•  Both  courses  have  a  two-dimensional  evaluation  program, 
subjective  student  end- of -course  and  instructor  or 
facilitator  evaluations.  Courses  are  periodically 
reviewed  by  the  course  proponent,  course  manager  and 
training  quality  assurance  representative.  Proponents 
provide  input  on  regulatory  and  policy  changes. 

•  We  conduct  post-graduation  surveys  with  graduates  and 
their  supervisors  biannually  to  assess  customer 
satisfaction. 

•  Surveys,  proponent  inputs,  MACOM  inputs. 

•  Input  from  field  or  functional  managers;  system  changes; 
data  provided  from  occupational  surveys. 

•  There  are  many  possibilities.  The  faculty  is  responsible 
for  maintaining  currency  in  their  field,  as  the  field 
changes,  so  does  the  course.  Other  sources  are  end- of - 
course  critiques,  post -course  critiques,  curricul\im 
reviews  and  communication  with  practitioners  and  sponsors. 

•  Current  issues  in  financial  management  plus  guidance  from 
senior  military  and  DoD  comptrollers. 

•  Mid-course  and  end-of -course  student  evaluations. 

•  In  order  to  determine  the  need  to  update  or  modify  the 
instruction  currently  offered,  we  test  alternative 
delivery  methods  of  instruction. 

•  Interaction  with  students,  instructional  staff  and  policy 
makers  from  respective  areas  of  subject  matter. 

•  Annual  course  reviews  and  periodic  curriculum  reviews 
conducted  by  fleet  or  type  commander  staff. 

•  Student  feedback,  major  claimant  input  along  with  changing 
issues  in  financial  management. 

•  End-of -course  critiques  and  surveys  of  past  graduates. 

•  Course  review;  changes  in  reference  material;  changes  in 
fleet  needs. 
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•  Continuous  review  of  instructors  and  student  critiques. 


•  The  main  determining  factor  for  modifying  our  instruction 
is  a  change  in  the  regulations  governing  financial 
management.  Post -graduation  feedback  questionnaires  and 
several  forms  of  internal  feedback  also  help  to  determine 
when  updates  need  to  be  made. 


*13 .  Do  you  have  one  or  more  sponsors  or  rUftrita  that 


directly  Influence  vour  orogy««i  r>f  Instruction? 


RESPONSE 

NUMBER 

PERCENTAGE 

YES 

18 

94.7 

NO 

1 

5.3 

TOTAL  RESPONSES 

19 

100.0 

If  ves.  who  Is/are  the  sponsor (s)? 


The  following  sponsors  were  specifically  mentioned  by 
survey  respondents: 

•  Secretary  of  the  Air  Force  for  Financial  Management 

•  Defense  Finance  and  Accounting  Service 

•  Chief  of  Naval  Operations 

•  Navy  Comptroller 

•  Chief  of  Naval  Education  and  Training 

•  Navy  Supply  Systems  Command 

•  Fiscal  Division,  Headquarters,  Marine  Corps 
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•  CINCIiANTFLT,  COMNAVAIRLANT ,  CNO 

•  Navy  Bureau  of  Personnel 

•  DoD  Comptroller 

•  Navy  Bureau  of  Medicine 

•  NAVFAC 

•  Defense  Acquisition  University 

•  NCD,  NCB,  CFMCP 

•  Navy  Exchange  Command 

•  Navy  Food  Service  Systems  Command 

*14.  Ifl  course  development  guided  or  directed  bv  higher 
authority? 


RESPONSE 

NUMBER 

PERCENTAGE 

YES 

13 

68.4 

NO 

6 

31.6 

TOTAL  RESPONSES 

19 

100.0 

If  80,  who? 

•  Soldier  Support  Center,  CASCOM,  TRADOC,  ASA  (FM) ,  DFAS 

•  Assistant  Secretary  for  the  Army  for  Financial  Management 

•  Corp  of  Engineers  Training  Issues  Committee 

•  Headquarters,  Air  Training  Command 

•  OP- 82/Director  of  Office  of  Budgets  and  Reports,  NCB 
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•  NCD,  NCB,  NCF 

•  Chief  of  Naval  Technical  Training 

•  CNET  and  NAVFAC 

•  BUMED 

•  COMNAVAIRLANT 

•  Standards  Branch,  Marine  Corps 

15.  What  types  of  do  vou  receive? 

The  following  comments  were  provided: 

•  Regulations  plus  directed  common  core  training. 

•  Learning  objectives. 

•  Feedback  from  collective  leadership  and  periodic 
assessment . 

•  Proponents  provide  guidance  on  and  approve  the  technical 
content . 

•  Regulations  and  policy  guidance. 

•  Subjects  of  concern  are  the  appropriate  content  for 
courses  and  the  levels  of  learning  required  for  each. 

•  General  guidelines  as  to  curriculum  content. 

•  Subject  matter  expertise, 

•  Selected  topic  material  or  requirements  for  graduate  skill 
levels. 

•  Course  review,  on-site  evaluations;  constant  feedback  is 
received  from  sponsors. 
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•  Curriculum  reviews,  directed  input,  issuance  of  new 
directives . 

•  Formal  instructions. 

•  Internal  curriculum  review. 

•  The  Marine  Corps  uses  the  Instructional  Systems 
Development  process  known  as  the  Systems  Approach  to 
Training  (SAT)  as  a  guideline  for  all  course  development. 

16 .  How  often  do  you  receive  such  cnildemce? 

The  typical  response  to  this  question  was  that  guidance 
was  received  whenever  necessary.  Fourteen  of  the  respondents 
answered  in  this  fashion.  Other  responses  were  annually  (1) , 
quarterly  (1) ,  and  two  to  three  times  a  year  (1) . 


*17 .  Do  you  have  staff  dedicated  tr?  pT‘9qir«m  development? 


RESPONSE 

NUMBER 

PERCENTAGE 

YES 

12 

66.7 

NO 

6 

33.3 

TOTAL  RESPONSES 

18 

100.0 

If  vea.  how 

The  answers  to  this  question  varied  from  one  to  fifty- 
eight  .  The  average  response  was  between  1  and  3 . 
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18. 


What  are  the  niiTnh«»-r  of  and  percentages  of  civilian 
and/or  military  Instructors  In  vour  orng-ram? 


Of  the  sixteen  agencies  providing  detailed  information  on 
this  question,  the  following  cumulative  data  was  obtained; 


CIVILIAN: 

Number 

142 

Percentage 

47.5 

MILITARY: 

Niuiber 

152 

Percentage 

52.5 

The  range  of  civilian  employees  was  from  0  to  66;  for 
military  employees  the  range  was  0  to  33.  The  average  (mean) 
number  of  civilian  employees  was  6.3  (one  agency  with  66 
civilian  employees  was  excluded  from  this  average) .  The 
average  (mean)  number  of  military  employees  was  13.1. 

19 .  What  policies  and  criteria  are  used  to  assign  or  hire 
military  faculty/ Ins true tors  for  vour  program? 


The  following  responses  were  provided: 


•  The  officer  or  enlisted  finance  assignment  branches  screen 
and  submit  records  of  the  best  available  personnel  to  the 
commandant  who  ma)ces  the  final  decision. 

•  Subject  matter  expertise,  previous  schooling,  previous 
assignments,  and  enlisted/officer  fitness  reports  are  all 
included  as  part  of  the  review  prior  to  actual  assignment. 

•  Grade,  educational,  experience  and  qualifications. 
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•  Military  candidates  are  screened  by  a  five  member  faculty 
committee. 

•  A  Master's  degree  is  required.  Broad  knowledge  of  the 
comptroller  field  with  broad  background  in  at  least  one 
functional  area.  Strong  performance  record. 

•  Must  have  graduate  degree  and  are  financial  management 
specialists . 

•  Must  have  subspecialty  code;  and  0-4  and  above. 

•  Manpower  efficiency  reviews  conducted  by  Chief  of  Naval 
Education  and  Training. 

•  Recent  academic  background  and  field  experience. 

•  NEC  9502. 

•  New  instructors  are  assigned  on  the  basis  of  their 
technical  knowledge  of  the  svibjer.t  matter  they  will  teach 
and  their  communication  skills. 


20.  What  policies  and  criteria  are  used  to  appoint  or  hire 
clvlU*"  faculty/ Ins  true  tors  for  vour  progyam? 

The  following  comments  were  provided: 


•  All  civil  service  positions  within  the  training  instructor 
field  have  prescribed  miniiaxim  qualifications. 

•  College  degree,  subject  matter  expertise,  ability  to 
teach. 

•  Coordination/consultation  with  department  chairs  and 
academic  directors. 

•  Instructors  are  recommended  by  their  supervisors. 

•  Review  of  job  series  and  job  experience.  Depending  on 
grade,  career  field  experience  is  generally  required. 


Final  selection  is  based  on  educational  requirements, 
svibject  matter  laiowledge,  experience,  interview,  and 
potential  to  perform  on  the  "platform" . 

•  Grade,  educational  experience,  qualifications  and  0PM 
guidance . 

•  Civilian  candidates  are  screened  by  a  five  member  faculty 
committee. 

•  Must  have  doctoral  degree.  For  senior  faculty,  must  also 
have  scholarly  research  record. 

•  Resumes,  references,  interviews. 

•  Level  of  degree,  professional  field  experience  in  finance. 


*21.  Are  professional  auallf Icatlona  standards  required  for 

facultv/lnatructors  (e.a.,  degrees,  prior  experience, 
completion  of  eg  inatructor  training  ? 


RESPONSE 

NUMBER 

PERCENTAGE 

YES 

17 

94.4 

NO 

1 

5.6 

TOTAL  RESPONSES 

18 

100.0 

If  ves.  Please  describe. 


•  College  degree. 

•  Same  standards  as  for  the  University. 

•  Most  instructors  have  a  degree  in  their  subject  area 
(minimtim  of  bachelor's)  and  not  less  than  five  years 
experience  in  their  subject  area.  Further,  instructors 
must  complete  the  Instructional  Methods  Course. 
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•  All  military  officers  have  degrees;  civilians  usually  have 
degrees.  The  Faculty  Development  Course  must  be  completed 
within  90  days  of  assignment. 

•  Associates  degree  or  higher;  completion  of  Technical 
Training  Instructor  Course. 

•  All  faculty  must  have  an  appropriate  Master's  degree  and 
experience  in  the  field  taught  or  a  closely  related  field. 
Faculty  hired  without  significant  prior  teaching 
experience  must  complete  Academic  Instructor  School . 

•  Master's  degree  plus  completion  of  Air  Force  Academic 
Instructor  School . 

•  We  prefer  faculty  with  Phd's  and  teaching  experience. 

•  Each  faculty  member  in  the  teaching  discipline  at  the 
master's  degree  level  will  hold  the  terminal  degree  in  the 
teaching  discipline  or  a  related  discipline.  Outstanding 
professional  experience  plus  a  Master's  degree  is 
preferred. 

•  Specific  Navy  enlisted  classification  codes  apply  to  some 
billets  which  require  fleet  experienced  personnel  to  be 
detailed  to  the  billet. 

•  Minimum  of  Bachelor,  desired  Master's  and  five  years  of 
experience . 

•  Instructor  Training  School;  subspecialty  designation. 

•  Graduate  of  instructor  training  school. 

•  Prior  experience  in  the  field. 

•  All  instructors  complete  a  two  week  Instructor  Training 
Course . 
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*22. 


Ss _ YOU  Tjise _ classroom  observation  to  evaluate 

Instruction? 


RESPONSE 

NUMBER 

PERCENTAGE 

YES 

17 

89.5 

NO 

2 

10.5 

TOTAL  RESPONSES 

19 

100.0 

If  ves.  who  performs  this  observation? 


The  following  responses  were  provided: 


•  Department  directors,  division  chiefs,  course  directors, 
commandant . 

•  Staff  and  faculty  development  representatives. 

•  Peers. 

•  Progrann  directors. 

•  Classroom  courses  are  evaluated  by  the  course  manager, 
quality  assurance  representative,  proponent  or  lead 
instructor. 

•  School  deans,  department  chairman,  course  directors. 

•  Instructor  supervisors  and/or  flight  commander. 

•  Department  head. 

•  Commandant. 

•  Regional  academic  director. 

•  Educational  specialists. 

•  Staff  instructors. 
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•  Academic  Standards  department. 

•  Academic  Program  division. 

•  Qualified  instructors. 

•  Training  director  and  department  head. 

•  Master  course  manager. 

•  Curriculum  Standards  Branch  Officer,  Instructional 
Operations  Officer,  Academic  Officer,  Academic  Chief, 
Instructional  Operations  Chief. 

How  often  Is  It  performed? 

The  following  responses  were  provided: 

•  Weekly  -  5  responses 

•  Quarterly  -  2  responses 

•  Twice  yearly  -  4  responses 

•  Annually  -  6  responses 

•  Unscheduled  or  as  appropriate  -  2  responses 


*23 .  Are  other  methods  used  to  evaluate  classroom 
Instruction? 


RESPONSE 

NUMBER 

PERCENTAGE 

YES 

15 

83.3 

NO 

*  3 

16.7 

TOTAL  RESPONSES 

18 

100.0 

If  ves.  Please  describe. 


Of  the  fifteen  agencies  which  responded  to  this  question, 
all  cited  student  critiques  as  a  method  used  to  evaluate 
classroom  instruction.  Detailed  responses  are  listed  below: 


e  student  evaluation  of  training. 

•  End- of -class  evaluation  by  students. 

•  Surveys  from  participants;  group  assessment  sessions. 

•  End- of -course  questionnaires  by  students;  audits  of  classes 
by  proponent  representatives;  post -graduation  surveys  sent 
to  graduates  and  their  supervisors  6  months  after  course 
conqpletion;  academic  excellence  analysis. 

•  ATC  form  736;  student  critiques;  Training  evaluation 
Reports;  Training  Quality  Reports  from  graduates  and 
supervisors  of  graduates;  field  visits;  STAN  evaluations 
and  Inspector  General  inspections. 

•  We  use  student/supervisor  post  course  critiques.  The 
student  class  leader  debriefs  the  department  head  at  the 
conclusion  of  each  course. 

•  Student  opinion  questionnaires. 

•  Mid-course  and  end-of -course  student  evaluations. 

•  Student  feedback  instrvunents;  discussions  with 
instructors/administrators . 

•  Student  feedback  from  every  class.  Representatives  from 
sponsor  also  monitor  classes. 

•  Student  critiques;  peer  review. 

•  Course  reviews;  student  critiques;  external  evaluations 
( surveys ) . 

•  Quarterly  improvement  form  completed  by  all  students. 
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•  Course  evaluation  upon  student  course  completion  and  a 
supervisor's  evaluation  of  student  performance  in  training 
skills  3  months  after  course  completion. 

•  Every  instructor  completes  an  after  instruction  report  to 
note  any  problems  with  lesson  materials.  Each  student 
fills  out  an  end- of -course  critique  prior  to  graduation 
which  enables  him  to  evaluate  the  overall  course  material . 


*24 .  Are  other  methods  used  to  evaluate  Instructors? 


RESPONSE 

NUMHER 

PERCENTAGE 

YES 

12 

63.2 

NO 

7 

36.8 

TOTAL  RESPONSES 

18 

100.0 

If  ves.  Please  describe. 


The  following  responses  were  received: 


•  Selection  of  instructor  of  the  month,  quarter  and  year. 

•  Survey  of  a  sample  of  students  and  their  supervisors 
conducted  six  months  after  course  completion  to  determine 
adequacy  and  application  made  of  training. 

•  End-of -course  questionnaires;  performance  appraisals; 
informal  visits  to  classroom  by  course  directors. 

•  Student  critiques;  STAN  evaluations;  Inspector  General 
inspections . 

•  The  student  course  critique  covers  the  instructor's 
performance . 
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•  student  opinion  questionnaires. 

•  Faculty  may  choose  a  15  minute  tape  presentation,  arrange 
a  supervisory  visit  and  evaluation  or  submit  a 
representative  portfolio  for  evaluation  purposes. 

•  Student  feedback  instruments.  Follow-up  discussions  with 
students/administrators . 

•  Student  critiques;  peer  review. 

•  Course  review;  student  critiques;  external  evaluations. 

•  Review  of  instructor  prepared  course  material . 

•  100  percent  of  the  students  of  each  class  complete  an 
Instructor  Rating  form.  This  form  allows  them  to  evaluate 
the  lesson  material  as  well  as  the  effectiveness  of  the 
instructor. 


*25.  Do  YOU  employ  productivity  measures  to  evaluate 
Instructors? 


RESPONSE 

ITtJMBSR 

PERCENTAGE 

YES 

3 

15.8 

NO 

16 

84.2 

TOTAL  RESPONSES 

19 

100.0 

If  so.  Please  describe. 

Only  three  of  the  nineteen  respondents  indicated  that 
productivity  measures  were  employed.  Of  these  three  only  one 
respondent  provided  measures  employed.  These  were; 


instructor  contact  hours;  student  training  years  per 
instructor;  instructor  contact  hours  per  student  training 
years  produced;  and  consulting  hours. 

*26.  Is  vour  Institution  accredited? 


SBSPONSE 

NUMBER 

PERCENTAGE 

YES 

9 

47.4 

NO 

10 

52.6 

TOTAL  RESPONSES 

19 

100.0 

If  ves.  bv  whom? 

•  North  Central  Association  of  Colleges  and  Universities, 
Commission  on  Institutions  of  Higher  Education. 

•  AACSB  for  Schools  of  Management. 

•  Southern  Association  of  Colleges  and  Schools  (2 
responses) . 

•  North  Central  Association. 

•  American  Council  on  Education  (2  responses) . 

•  Western  Association  of  Schools  and  Colleges. 

•  Middle  States  Association. 

How  often? 

Frequency  varied  from  annually  to  every  ten  years. 
Typically,  the  review  cycle  is  five  years. 
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*27 .  Is  vour  institution  reviewed  bv  other  agencies? 


RESPONSE 

NUMBER 

PERCENTAGE 

YES 

13 

68.4 

NO 

6 

31.6 

TOTAL  RESPONSES 

19 

100.0 

If  ves.  bv  whom  often? 

Responses  received  included: 


•  Army  Training  and  Doctrine  Command. 

•  American  Council  on  Education  (2  responses) . 

•  Defense  Finance  and  Accounting  Agency;  Secretary  of  the 
Air  Force  for  Financial  Management;  Air  Force  Audit 
Agency;  Community  College  of  the  Air  Force. 

•  Air  University  (2  responses) . 

•  National  Association  of  Schools  of  Public  Affairs  and 
Administration. 

•  MIVER  and  QES. 

•  Commander,  Training  Command,  U.  S.  Atlantic  Fleet. 

•  CNET;  CNTECTRA;  NAVFAC. 

•  Navy  and  Bureau  of  Medicine  Inspector's  General. 

•  Type  commander. 

•  Command  inspections. 
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28 .  What  Is  the  nature  and  extent  of  this  review? 


The  following  responses  were  provided: 


•  Standard  ACE  review  of  course  material  and  lesson  plans. 

•  Courses  and  library  support  are  evaluated  by  ACE  teams  of 
subject  matter  experts.  Evaluators  consider  factors  such 
as  course  objectives,  siibject  matter,  level  of  difficulty, 
duration,  methods  of  student  evaluation,  applicability  to 
post -secondary  education  programs,  and  background  and 
selection  of  students  and  instructors.  ACE  then  makes 
recommendations  to  colleges/universities  for  granting 
academic  credit. 

•  To  determine  accuracy  and  currency  of  training. 

•  Primarily  focuses  on  management. 

•  Curriculum  review. 

•  Year-long  self-assessment  followed  by  site  visit. 

•  Formal  Inspector  General  inspections. 

•  Review  of  curriculum  and  adherence  to  directives. 

•  Inspections. 
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29 


vour  p-rnq-ram  (e.a..  lecture,  discussion,  simulation. 
Independent  study) ? 


MODE  OF  INSTRUCTION 

NUMBER 

PERCENT 

LECTURE 

16 

94.1 

DISCUSSION 

10 

58.8 

SIMULATION 

8 

47.1 

CASE  STUDY 

7 

41.2 

SEMINARS 

2 

11.8 

INDEPENDENT  STUDY 

6 

35.3 

PROJECT  TEAMS 

3 

17.6 

FIELD  TRIPS 

1 

5.9 

LABORATORIES 

2 

11.8 

PERFORMANCED  BASED 

2 

11.8 

The  typical  responses  to  this  question  included  lectures, 
discussions,  case  studies,  simulation  and  self  or  independent 
study.  All  respondents  save  one  cited  lectures  as  the 
predominant  mode  of  instruction. 
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mode  you  Identified? 

Of  the  eleven  institutions  providing  detailed  percentages 
on  modes  of  instruction,  all  cited  lectures  and  discussions  as 
the  predominant  modes  utilized.  On  average,  lectures 
represented  40  percent  of  the  instructional  program; 
discussion  represented  20  percent.  Simulation,  case  study  and 
independent  study  represent  approximately  10  percent  each. 
The  remaining  10  percent  was  divided  among  the  modes  reported 
in  Question  29  above. 

31.  What  types  of  course  reading  are  used  In 

vour  pyftq-r<itn  (e.q..  textbooks,  published  articles. 
DoD  official  dor-inn«Ti<-«.  Self -generated  materials)? 

Sixteen  agencies  responded  to  this  question.  Thirteen 


cited  DoD  official  documents.  Ten  developed  their  own  (self- 
generated)  materials.  And  nine  utilized  textbooks. 


*32. 


Is  vour  proarawi  anpported  bv  a  library  containing 
extensive  DoD  financial  management  related 


RESPONSE 

NUMBER 

PERCENTAGE 

YES 

9 

50.0 

NO 

9 

50.0 

TOTAL  RESPONSES 

18 

100.0 

Of  the  eighteen  responses  received,  half  noted  that  they 
were  not  adequately  supported  by  a  library  containing 
extensive  DoD  financially  related  materials.  Question  33  and 
34  below  ask  specific  question  on  library  support.  Six 
providers  specified  improvements  that  would  like  to  see  in 
library  support  (See  Question  33)  and  six  noted  that 
improvements  were  planned  in  library  support  (See  Question 
34)  .  The  issue  not  quantified  in  these  series  of  questions  is 
the  importance  or  lack  of  importance  to  the  program  of 
adequate  library  support. 
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33 


program? 

Six  responses  were  provided  on  this  question.  They 
included: 

•  Establish  a  library  within  our  school. 

•  Be  able  to  interact  with  some  DoD  instructional  material 
data  base. 

•  Ensure  that  the  library  has  all  required  materials. 

•  Improve  library  support  by  joining  the  Washington  area 
consortium. 

•  Specified  funds  for  financial  texts. 

•  Obtain  sufficient  copies  of  reference  materials. 


*34 .  Are  there  Improvements  In  vour  library  services 


RESPONSE 

NUMBER 

PERCENTAGE 

YES 

6 

35.3 

NO 

11 

64.7 

TOTAL  RESPONSES 

17 

100.0 
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If  ves.  Please  explain  theg**  ImpT-nv^menha  and 


Indicate  when  they  will  occur. 


•  An  effort  is  underway  to  automate  the  Logistics  Library 
which  will  provide  computer  assisted  in-house  services 
plus  remote  dial-in  access.  (Scheduled  for  FY  1993) 

•  Our  library  has  just  assigned  collection  development 
specialists  to  enhance  library  support  for  school 
programs . 


*35.  Is  vour  p-rriq-rjiTii  ijiypported  bv  a  computer  center  or 
laboratory? 


RESPONSE 

NUMBER 

PERCENTAGE 

YES 

14 

73,7 

NO 

5 

26.3 

TOTAL  RESPONSES 

19 

100.0 

If  no.  what  computer  resources  are  employed  at  vour 
Institution. 

Only  one  response  was  provided  in  response  to  this  portion 
of  the  question;  that  response  indicated  that  only  personal 
computers  were  used. 
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*36. 


Does  vour  Instructional  proqyam  include  computer 
exercises  or 


RESPONSE 

ITUMBER 

PERCENTAGE 

YES 

16 

88.9 

NO 

2 

11.1 

TOTAL  RESPONSES 

18 

100.0 

If  ves.  Please  describe  how  computers  are  utilized. 

Responses  Included: 

•  Standard  Army  Information  Management  System. 

•  Real-world  applications. 

•  Familiarize  clients  with  computers. 

•  Simulation. 

e  Data  retrieval  exercises;  National  Budget  case. 

•  Self-paced  instruction. 

•  Application  practices  (4  responses) . 
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RESPONSE 

NUMBER 

PERCENTAGE 

YES 

12 

63.2 

NO 

7 

36.8 

TOTAL  RESPONSES 

19 

100.0 

What  Improvements  In  support  wo\xld  be  desirable? 

Of  the  four  agencies  that  specified  specific  improvements, 
all  noted  additional  funding  for  guest  speakers  would  be 
desirable. 


38 .  What  methods  are  used  to  determine  the  need  for  vour 
current  p-r«<yrjtfny 

The  following  responses  were  provided; 

•  The  Army  conducts  annual  surveys  of  all  users  of  Army 
training  programs. 

•  Sponsors. 


•  Needs  assessments  are  conducted  by  proponents  prior  to  any 
consideration  of  bourse  design  or  development.  Proponents 
annually  verify  the  need  for  classroom  courses  and  an 


annual  market  survey  provided  the  needs  verification  from 
the  users. 

•  The  Total  Army  Centralized  Individual  Training 
Solicitation  (TACITS)  is  a  survey  which  is  conducted 
annually  for  primary  information  and  follow-up;  Structure 
Manning  Decision  Review  is  conducted  annually;  the 
Training  Resources  Arbitration  Panel  (TRAP)  process  is 
done  monthly;  off-line  memorandvims  can  be  submitted  once 
the  TRAP  closes  for  execution;  curriculum  council  reviews; 
annual  course  proponent  reviews. 

•  Field  evaluations;  Training  Evaluation  reports;  Training 
Review  Analyses;  occupational  surveys. 

•  Curriculum  review  addresses  this  specific  issue.  The  need 
is  also  assessed  by  the  quantity  of  requests  for  any 
particular  course. 

•  The  need  for  resource  management  education  among  military 
services. 

•  Data  on  P- coded  billets  and  other  financial  management 
positions . 

•  Program  reviews  are  conducted  every  five  years. 

•  Alumni  surveys;  student  surveys;  end- of -course 
assessments;  ESO  inputs;  and  faculty  surveys. 

•  NC  and  NCD  guidance;  Career  Program  Planning  Board; 
initiatives  in  the  operational  environment. 

•  Site  surveys  conducted  by  systems  command  sponsors  and 
annual  course  curriculum  reviews. 

•  Student  feedback  and  sponsors'  requirements. 

•  Demand  for  student  placements. 

•  Fleet  need;  tasking  from  higher  authority;  student 
critiques . 

•  Review  by  claimant. 

•  Extensive  course  review  by  program  manager  and  field 
personnel . 
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•  The  Marine  Corps  uses  the  Individual  Training  Standards 
that  describe  the  specific  tasks  and  knowledge  required. 
We  also  conduct  Course  Content  Review  Boards  to  ensure  the 
training  standards  are  correct. 


39 .  What  methods  are  used  to  determine  the  validity. 

accuracy.  ♦;<meiiness  of  vour  current 


The  following  methods  were  cited: 


•  Annual  review  of  training  strategy  by  subject  matter 
experts;  annual  review  of  lesson  plans;  needs  analysis. 

•  Proposed  training  program  based  on  TACITS  data  is  reviewed 
and  approved  by  the  Department  of  the  Army. 

•  Accreditation  and  standards  of  graduate  program. 

•  Proponents  are  charged  with  the  responsibility  for 
technical  content.  Proponent/instructor  meetings  as  well 
as  monitoring  of  courses  are  performed  to  ensure  technical 
accuracy,  timeliness,  etc.  Student  end- of -course 
critiques  are  constantly  monitored  for  student  perceived 
requirements  for  change. 

•  Curriculum  council  reviews;  annual  course  proponent 
reviews;  end- of -course  questionnaires;  Structure  Manning 
Decision  review;  Army  Training  Resource  Requirements 
System. 

•  Field  evaluations;  Training  Evaluation  Reports;  Training 
Review  Analysis;  occupational  surveys;  Inspector  General 
inspections;  student  critiques.  Also  a  customer  service 
information  line  has  been  installed. 

•  Curriculum  reviews;  visits  with  the  sponsor;  student 
critiques;  functional  boards;  and  other  feedback. 
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•  Review  by  senior  financial  management  officials;  student 
feedback. 

•  Observation  of  changes  in  governmental  and  academic 
practices  and  problems.  Consultation  with  program 
sponsor. 

•  Surveys  and  interviews  with  students,  alumni  and 
practitioners  in  the  field. 

•  All  programs  are  under  continuous  review.  Inputs  from  the 
teaching  site  is  gathered  each  term  to  determine  the 
validity  of  our  current  program. 

•  Subject  matter  experts;  career  board  annual  report, 
classroom  visits,  surveys  of  the  community. 

•  Annual  course  reviews;  systems  command  assessments  and 
technical  audits. 

•  Sponsor  review;  willingness  of  users  to  reimburse  for  the 
course . 

•  Curriculum  survey  to  past  graduates;  review  by  sponsors. 

•  Course  reviews;  standardization  conferences. 

•  Student /claimant  feedback. 

•  Competent  job  performers  and  subject  matter  experts  review 
course  content  for  validity,  accuracy  and  relevance. 

•  Following  completion  of  each  course,  a  career  content 
review  board  is  conducted  to  ensure  the  material  is 
accurate,  timely  and  relevant. 
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40. 


What  methods  are  Ampinyed  to  determine  student 


competence  during  and  upon  program  fr»iBt>letlon  (e.a.. 
passing  standardized  tests.  course  work, 

observation  of  performance  on  the  lob)? 

Responses  provided  included: 

•  Practical  exercises  (4  responses) 

•  Tests  (14  responses) 

•  Research  papers  (4  responses) 

•  Individual  or  group  projects  (3  responses) 

•  Case  studies  (4  responses) 

•  Oral  presentations  (3  responses) 

•  Passing  national  excuns  (2  responses) 

•  Thesis  (1  response) 

*41.  Do  you  have  a  formal  method  of  student  course 
evaluation? 


RESPONSE 

NUMBER 

PERCENTAGE 

YES 

19 

100.0 

NO 

0 

0.0 

TOTAL  RESPONSES 

19 

100.0 
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If  ves.  describe  this  process. 


Of  the  nineteen  responses  received  to  this  question,  all 
indicated  that  a  formal  method  of  student  course  evaluation 
existed.  All  used  a  student  critique  form  for  this  purpose. 


*42 .  Are  other  methoda  used  to  permit  students  to 
evaluate  vour  Instructional  prrxyrjim? 


RESPONSE 

NUMBER 

PERCENTAGE 

YES 

10 

52.6 

NO 

9 

47.4 

TOTAL  RESPONSES 

19 

100.0 

If  vee.  describe  these  methods. 

The  following  comments  were  provided: 

•  The  Commandant  holds  periodic  informal  discussions  with 
students. 

•  Post -graduation  surveys  (3  responses) . 

•  DD  form  1556  is  completed  by  students  and  their 
supervisors . 

a  Student- to- instructor  feedback. 
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•  Students  provide  informal  comments  to  their  class  leaders, 
who  can  relay  them  at  end- of -course  debriefs. 

•  Students  are  invited  to  meet  with  the  department  chairman 
periodically  and  can  make  individual  comments  to  him  as 
they  wish. 

•  Follow-up  with  educational  specialists. 


*43 .  Are  your  students  required  to  paaa  any  «t--*ndardlzed 

DoD  or  national  «»-if^™inatlons  (e.a..  CPA.  CMA)  ? 


RESPONSE 

NUMBER 

PERCENTAGE 

YES 

1 

5.3 

NO 

18 

94.7 

TOTAL  RESPONSES 

19 

100.0 

The  one  respondent  answering  "yes"  to  this  indicated 
that  students  are  required  to  pass  the  GRE,  GMAT  or  MAT. 

44 .  What  percentage  of  students  entering  vour  prooMm 
successfully  complete  It? 

Of  the  eleven  responses  to  this  question,  10  indicated 
that  completion  rates  averaged  well  over  95  percent.  One 
agency  cited  a  completion  rate  of  75  percent. 
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RESPONSE 

NUMBER 

PERCENTAGE 

YES 

7 

36.8 

NO 

12 

63.2 

TOTAL  RESPONSES 

19 

100.0 

If  ves.  how  la  thla  done? 


Of  the  seven  responses  provided,  six  agencies  cited  the 
use  of  post-graduate  surveys  as  the  principle  method  utilized 
to  track  graduate  performance.  Specific  comments  were: 


•  School  liaison  teams  visit  finance  units  in  the  field  to 
evaluate  school  training  and  training  needs  with  graduates 
and  their  supervisors.  Survey  of  a  sample  of  students  and 
their  supervisors  is  conducted  about  six  months  after 
course  completion  to  determine  adequacy  and  applicability 
of  training. 

•  Post  graduate  surveys  are  sent  to  graduates  and  their 
supervisors  six  months  after  course  completion. 

•  Field  visits  by  training  evaluators. 

•  We  use  one  year  and  five  year  out  surveys  of  graduates. 

•  Follow-up  questionnaire  one  year  after  completion;  survey 
of  supervisors  of  graduates  of  our  program;  track 
progression  through  the  alumni  association. 

•  A  post -course  evaluation  is  sent  to  the  student's 
supervisor.  It  is  intended  to  determine  the  effectiveness 
of  training  more  than  the  performance  of  the  individual . 
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•  The  supervisors  of  all  graduates  are  sent  a  feedback 
questionnaire  to  help  us  evaluate  if  the  student  learned 
the  necessary  prerequisite  knowledge. 


*46 .  Are  formal  mAthoHa  «wtpi<;>Y«*d  to  solicit  feedback  from 
former  fiiAnha  ^v>r>iit  the  utility  of  vour 

program? 


RESPONSE 

NUMBER 

PERCENTAGE 

YES 

12 

63.2 

NO 

7 

36.8 

TOTAL  RESPONSES 

19 

100.0 

If  vea.  how  la  this  done? 

Of  the  twelve  responses  received,  nine  agencies  cited  the 
use  of  post-graduate  surveys  for  this  purpose.  Post-graduate 
surveys  were  used  for  two  purposes:  (1)  To  track  the 
performance  of  graduates  (see  Question  45  above) ,  and  (2)  To 
assess  the  effectiveness  or  utility  of  the  program. 

Post-graduate  surveys  are  extensively  discussed  in  Chapter 

IV. 
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47. 


In  vour  opinion,  what  are  the  most  Important  methods 
you  aacDlov  to  ensure  the  quality  of  von-r  prog-ram? 


This  question  was  designed  to  be  the  cap-stone  question  of 
the  survey.  It  summarizes  the  methods  used  by  survey 
respondents  to  ensure  quality.  Responses  included: 


•  Visits  to  finance  units  in  the  field;  selection  of 
instructors/subject  matter  experts  who  have  field 
experience;  insuring  that  we  are  kept  informed  on  the 
numerous  changes  in  finance,  accounting  and  resource 
management  regulations  and  policies. 

•  Qualified  faculty;  feedback  from  students  and  their 
supervisors . 

•  Recognition  by  sponsors  and  commands/agencies  which 
provide  participants  for  the  program;  competition  for 
individuals  to  attend;  long  standing  reputation  of  the 
program, 

•  Needs  assessment;  task  analysis;  analysis  of  pre/post  test 
results;  quality  assurance;  proponent  and  course  manager 
evaluations;  periodic  proponent /ins true tor /SME  workshops 
for  technical  accuracy  and  relevance;  analysis  of  student 
end- of -course  evaluations. 

•  Close  attention  to  post -graduation  surveys;  Close 
attention  to  comments  on  end- of -course  questionnaires; 
continuing  dialogue  with  students  and  others  in  the 
functional  area;  exceptional  instruction;  extensive 
knowledge  of  subject  matter  and  regulatory  requirements. 

•  Field  visits;  student  evaluations;  rotation  of  military 
instructors;  field  feedback;  workshops  to  review  training. 

•  We  attempt  to  hire  and  maintain  a  top  quality  faculty. 
The  Faculty  Academics  Standards  Committee  reviews  each 
course  every  three  years . 
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•  Constant  review  of  the  curriculum  by  the  faculty  plus 
biannual  review  by  Air  University  and  senior  DoD 
comptrollers;  student  feedback. 

•  Faculty  involvement  with  the  Navy  and  Defense  Department 
and  in  academic  activities  and  organizations. 

•  We  set  high  standards  for  professional  graduate  level  work 
and  expect  the  faculty  to  meet  these  standards. 

•  Student  course  assessments;  faculty  observations. 

•  Review  of  material  for  accuracy  of  content;  student  and 
instructor  evaluations;  post -training  job  performance. 

•  Periodic  fleet/type  commander  staff  reviews  and  systems 
command  technical  assessments. 

•  Constant  feedback  from  the  students  and  inputs  from  the 
course  sponsor. 

•  Collection  of  data  via  student  critiques;  trend  analysis 
over  time. 

•  Fleet  inputs. 

•  On-going  student  and  client  feedback. 

•  Feedback  from  students  and  their  supervisors. 

•  The  Instructional  Systems  Development  Process  guides  us  to 
ensure  the  quality  of  our  program.  Additionally,  our 
school  enjoys  an  outstanding  relationship  with  field 
personnel.  We  are  able  to  respond  to  changes  to 
regulations  and  provide  better  instruction. 


48 .  Is  there  any  other  Information  that  vou  would  offer  on 

quality  assessment  at  vour  Institution  that  we  did  not 
request? 


No  responses  were  provided  to  this  question. 


116 


LIST  OF  REFERENCES 


1.  Wildavsky,  Aaron,  The  New  Politics  of  the  Budgetary 

Process.  Scott,  Foresman  and  Company,  1988. 

2.  Executive  Office  of  the  President,  Office  of  Management 

and  Budget,  Federal  Budget  for  Fiscal  Year  1993. 

Government  Printing  Office,  January,  1992. 

3.  Comptroller  of  the  Department  of  Defense,  Career 
Structures  Work  Group  Report.  August,  1991. 

4.  Executive  Office  of  the  President,  Office  of  Management 

and  Budget ,  Federal  Budget  for  Fiscal  Year  1992. 

Government  Printing  Office,  January,  1991. 

5.  Defense  Management  Report  Decision  No.  985,  Financial 

Management  Education  and  Training.  Department  of  Defense. 
(Fiscal  Year  1991) 

6.  Defense  Management  Report  Decision  No.  985,  Financial 

Management  Education  and  Training.  Department  of  Defense. 
(Fiscal  Year  1992) 

7.  Office  of  the  Comptroller  of  the  Department  of  Defense 
Memorandum,  Management  Plan  for  the  Review  of  Financial 
Management  Education  and  Training.  June  3,  1991. 

8 .  Department  of  Defense  Training  and  Performance  Data 

Center,  Formal  Schools  Directory.  January,  1992. 

9 .  Department  of  Defense  Training  and  Performance  Data 

Center,  Financial  Management  Data  System.  January,  1992. 

10.  Berk,  Ronald  A. ,  editor.  Education  Evaluation  Methodology: 
The  State  of  the  Art.  John  Hopkins  University  Press,  1981. 

11.  Biemer,  Paul  P.  et  al.  Measurement  Errors  in  Surveys.  John 
Wiley  and  Sons,  Inc.,  1991. 


12.  Brandt,  Ronald  S.,  editor.  Applied  Strategies  for 
Curriculum  Evaluation.  Association  for  Supervision  and 
Curriculxom  Development,  1981. 

13.  Doll,  Ronald  C.,  Curriculum  Improvement:  Decision  Making 
and  Process.  Allyn  and  Bacon,  1992. 

14.  Dressel,  Paul  L.,  Handbook  of  Academic  Evaluation.  Jossey- 
Bass  Publishers,  1978. 

15.  English,  Fenwick  W. ,  Curriculum  Management  for  Schools. 
Colleges.  Businesses.  Charles  C.  Thomas  Publishers,  1987. 

16.  Euske,  Kenneth  J. ,  Management  Control:  Planning.  Control. 
Measurement  and  Evaluation.  Addison -Wesley  Publishing 
Company,  1984. 

17.  Ewell,  Peter  T. ,  "To  Capture  the  Ineffable:  New  Forms  of 
Assessment  in  Higher  Education",  Review  of  Research  in 
Education. 

18.  Rossi,  Peter  H. ,  et  al.  Evaluation:  A  Systematic  Approach. 
Sage  Pixbli  cat  ions,  1979. 

19.  Stufflebeam,  Daniel  L.,  et  al.  Education  Evaluation  and 
Decision  Making.  F.  E.  Peacock  Publishers,  1971. 

20.  Suchman,  Edward  A.,  Evaluation  Research:  Principles  and 
Practices  in  Public  Service  and  Social  Action  Prograi 
Russell  Sage  Foundation,  1967. 

21.  Weiss,  Carol  H.,  Evaluation  Research:  Methods  of  Assess: 
Program  Effectiveness.  Prentice - Hall ,  Inc.,  1972. 


22.  Ewell,  Peter  T. ,  letter  to  LT.  Michael  M.  Gragen  of 
February  5,  1992. 


INITIAL  DISTRIBUTION  LIST 


No.  Copies 

1.  Defense  Technical  Information  Center  .  2 

Cameron  Station 

Alexandria,  VA  22304-6145 

2.  Library,  Code  0142 . 2 

Naval  Postgraduate  School 

Monterey,  CA  93943-5002 

3.  Professor  Kenneth  Euske,  Code  AS/EE  .  l 

Naval  Postgraduate  School 

Monterey,  CA  93943-5002 

4.  Professor  Larry  Jones,  Code  AS/JN . 1 

Naval  Postgraduate  School 

Monterey,  CA  93943-5002 

5.  Professor  Jerry  McCaffrey,  Code  AS/MC  .  1 

Naval  Postgraduate  School 

Monterey,  CA  93943-5002 

6.  Mr.  Peter  T.  Ewell  . . 1 

Senior  Associate 

National  Center  for  Higher  Education  Management 
P.  0.  Drawer  P 
Boulder,  CO  80301-9752 

7.  CDR  Dennis  Allion,  Code  36 . l 

Naval  Postgraduate  School 

Monterey,  CA  93943-5000 

8.  LT  Michael  M.  Gragen . 1 

202  Thames  Street 

New  London,  CT  06320 


119 


